Image processing apparatus, image processing method, electronic equipment and program

ABSTRACT

The present technology relates to an image processing apparatus, an image processing method, electronic equipment and a program which can remove cyclic noise from an image including the cyclic noise. 
     An estimating unit configured to estimate cyclic noise components included in each image picked up under different exposure conditions for each image is included. The estimating unit estimates the cyclic noise components for each image through operation utilizing mutual relationship between the noise components under the exposure conditions. For example, the cyclic noise is flicker. The mutual relationship between the noise components may be expressed with a shutter function of the exposure conditions in frequency space. The present technology may be applied to an imaging apparatus.

TECHNICAL FIELD

The present technology relates to an information processing apparatus,an information processing method, electronic equipment and a program.Specifically, the present technology relates to an image processingapparatus, an image processing method, electronic equipment and aprogram for correcting flicker occurring in an image.

BACKGROUND ART

When an image is picked up using a camera equipped with an XY-addressscan type imaging device such as a complementary metal oxidessemiconductor (CMOS) imaging device under illumination of a fluorescentlight, stripe brightness unevenness or color unevenness occurs in avideo signal. This phenomenon is referred to as flicker. This flicker iscaused due to a fluorescent light connected to a commercial power supply(AC) repeating blinking basically with a cycle double a cycle of apower-supply frequency and due to operating principle of the imagingdevice.

A stripe brightness change pattern extending in a horizontal directionappears in an image in which flicker occurs. For example, when a movingimage is observed, a stripe pattern appearing vertically is observed.Examples of related art which discloses a technique for suppressing suchflicker include, for example, Patent Literature 1. Patent Literature 1discloses a method for removing a flicker component included in an imageby extracting the flicker component from the image, calculating aflicker correction coefficient which has a reversed-phase pattern of theflicker component and performing correction by multiplying a pixel valueof the image by the flicker correction coefficient.

CITATION LIST Patent Literature

Patent Literature 1: JP 2011-160090A

SUMMARY OF INVENTION Technical Problem

By the way, for example, in order to generate a high dynamic rangeimage, an imaging apparatus has been proposed which generates a highdynamic range image for which a more accurate pixel value is set from alow brightness portion to a high brightness portion by picking up aplurality of images for which different exposure periods are set.

When the above-described processing disclosed in Patent Literature 1 istried to be applied to the imaging apparatus which picks up theplurality of images in different exposure periods as described above, itis necessary to execute processing such as processing of extracting aflicker component, processing of calculating a reversed-phase correctioncoefficient of the flicker component and correction processing based onthe correction coefficient individually for each of the plurality ofimages picked up in different exposure periods.

In this manner, in order to execute the above-described processing oneach of the images with different exposure periods, there is apossibility that hardware components may increase and a processingperiod may increase.

The present technology has been made in view of such circumstances, andis directed to making it possible to efficiently execute processing ofreducing cyclic noise such as flicker with a simple configuration.

Solution to Problem

According to an aspect of the present technology, an image processingapparatus includes: an estimating unit configured to estimate cyclicnoise components included in each image picked up under differentexposure conditions for each image. The estimating unit estimates thecyclic noise components for each image through operation utilizingmutual relationship between the noise components under the exposureconditions.

The cyclic noise may be flicker.

The mutual relationship between the noise components may be expressedwith a shutter function of the exposure conditions in frequency space.

The estimating unit may estimate the noise components using a valueobtained by integrating the images, multiplying a predetermined windowfunction and performing Fourier series expansion.

The integration may be performed in a horizontal direction for a portionnot saturated in any of the images.

The estimating unit may obtain the noise components in frequency spaceby obtaining a matrix Q where QF=0 when a fluctuation component of alight source is F and obtaining the fluctuation component F. Theestimating unit may estimate the noise components for each image byperforming Fourier series inverse transform on the noise components inthe frequency space.

The mutual relationship between the noise components may be expressedwith a ratio obtained by integrating the images and performing divisionfor each row of the images.

The integration may be performed in a horizontal direction for a portionnot saturated in any of the images.

The estimating unit may obtain an eigenvector of an eigenvalue 1 of amatrix RT where R is a matrix obtained by performing Fourier seriesexpansion on the ratio and T is a matrix obtained from the exposureconditions, and set the eigenvector as a value obtained by performingFourier series expansion on the noise components of the images.

The noise components of the images may be calculated by performingFourier series inverse transform on the eigenvector.

A value obtained by performing Fourier series expansion on the noisecomponents of an image different from an image for which the noisecomponents have been calculated may be calculated by multiplying theeigenvector by a coefficient obtained from the exposure conditions. Thenoise components of the images may be calculated by performing Fourierseries inverse transform on the value obtained by performing Fourierseries expansion.

The estimating unit may generate a matrix RT in the following formula,

$\begin{matrix}{\left\lbrack \begin{matrix}R_{0} & {\overset{\_}{R}}_{1} & \ldots & {\overset{\_}{R}}_{M} & \; & \; & 0 \\R_{1} & R_{0} & \ddots & \ddots & \ddots & \; & \; \\\vdots & \ddots & R_{0} & {\overset{\_}{R}}_{1} & \ddots & \ddots & \; \\R_{M} & \ddots & \ddots & R_{0} & \ddots & \ddots & {\overset{\_}{R}}_{M} \\\; & \ddots & \ddots & R_{1} & R_{0} & \ddots & \vdots \\\; & \; & \ddots & \ddots & \ddots & R_{0} & {\overset{\_}{R}}_{1} \\0 & \; & \; & R_{M} & \ldots & R_{1} & R_{0}\end{matrix} \right\rbrack {\quad{{\left\lbrack \begin{matrix}{\overset{\_}{T}}_{M} & \; & \; & \; & \; & \; & \; \\\; & \ddots & \; & \; & \; & 0 & \; \\\; & \; & {\overset{\_}{T}}_{1} & \; & \; & \; & \; \\\; & \; & \; & T_{0} & \; & \; & \; \\\; & \; & \; & \; & T_{1} & \; & \; \\\; & 0 & \; & \; & \; & \ddots & \; \\\; & \; & \; & \; & \; & \; & T_{M}\end{matrix} \right\rbrack \begin{bmatrix}\overset{\_}{G_{1}(M)} \\\vdots \\\overset{\_}{G_{1}(1)} \\{G_{1}(0)} \\{G_{1}(1)} \\\vdots \\{G_{1}(M)}\end{bmatrix}} = \begin{bmatrix}\overset{\_}{G_{1}(M)} \\\vdots \\\overset{\_}{G_{1}(1)} \\{G_{1}(0)} \\{G_{1}(1)} \\\vdots \\{G_{1}(M)}\end{bmatrix}}}} & \left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack\end{matrix}$

where R is the ratio, T is the coefficient obtained from the exposureconditions and G is the noise components of the images, and obtain thenoise components of the images.

The estimating unit may obtain an eigenvector of an eigenvalue 1 of amatrix rt where a matrix r is the ratio and a matrix t is a matrixobtained from the exposure conditions, and estimate that the eigenvectoris the noise components of the images.

The noise components of an image different from an image for which thenoise components have been calculated may be calculated from a linearsum of the estimated noise components.

The estimating unit may obtain the noise components for each image byobtaining g₁, g₂ which satisfy the following formula throughleast-squares estimation,

$\begin{matrix}{{\begin{bmatrix}t & {- I} \\I & {- r}\end{bmatrix}\begin{bmatrix}g_{1} \\g_{2}\end{bmatrix}} = 0} & \left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack\end{matrix}$

where r is the ratio, t is the value obtained from the exposureconditions, I is a pixel value of the images and g is the noisecomponents.

According to an aspect of the present technology, an image processingmethod includes: an estimating step of estimating cyclic noisecomponents included in each image picked up under different exposureconditions for each image. The estimating step includes processing ofestimating the cyclic noise components for each image through operationutilizing mutual relationship between the noise components under theexposure conditions.

According to an aspect of the present technology, a program causing acomputer to execute processing includes: an estimating step ofestimating cyclic noise components included in each image picked upunder different exposure conditions for each image. The estimating stepincludes processing of estimating the cyclic noise components for eachimage through operation utilizing mutual relationship between the noisecomponents under the exposure conditions.

According to an aspect of the present technology, electronic equipmentincludes: a signal processing unit configured to perform signalprocessing on a pixel signal outputted from an imaging device. Thesignal processing unit includes an estimating unit configured toestimate cyclic noise components included in each image picked up underdifferent exposure conditions for each image, and a correcting unitconfigured to perform correction to remove noise from the images usingthe noise components estimated at the estimating unit. The estimatingunit estimates the cyclic noise components for each image throughoperation utilizing mutual relationship between the noise componentsunder the exposure conditions.

In the image processing apparatus, the image processing method and theprogram according to one aspect of the present technology, cyclic noisecomponents respectively included in images picked up under differentexposure conditions are estimated for each image. The estimation isperformed by estimating a cyclic noise component for each image throughoperation utilizing mutual relationship between noise components underexposure conditions.

In the electronic equipment according to one aspect of the presenttechnology, signal processing is performed on a pixel signal outputtedfrom the imaging device, and as one processing of the signal processing,cyclic noise components respectively included in images picked up underdifferent exposure conditions are estimated for each image, andcorrection is performed to remove noise from the images using theestimated noise components. The estimation is performed by estimating acyclic noise component for each image through operation utilizing mutualrelationship between noise components under exposure conditions.

Advantageous Effects of Invention

According to one aspect of the present technology, it is possible toefficiently execute processing of reducing cyclic noise such as flickerwith a simple configuration.

It should be noted that the advantageous effects of the presentinvention are not limited to the advantageous effects described herein,and may include any advantageous effect described in the presentdisclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for explaining occurrence principle and correctionof flicker.

FIG. 2 is a diagram for explaining occurrence principle and correctionof flicker.

FIG. 3 is a diagram for explaining occurrence principle and correctionof flicker.

FIG. 4 is a diagram illustrating a configuration of an embodiment of animaging apparatus to which the present technology is applied.

FIG. 5 is a diagram illustrating an arrangement example of pixels withdifferent exposure periods.

FIG. 6 is a diagram illustrating an arrangement example of pixels withdifferent exposure periods.

FIG. 7 is a diagram illustrating an arrangement example of pixels withdifferent exposure periods.

FIG. 8 is a diagram illustrating an arrangement example of pixels withdifferent exposure periods.

FIG. 9 is a diagram illustrating an arrangement example of pixels withdifferent exposure periods.

FIG. 10 is a diagram illustrating an arrangement example of pixels withdifferent exposure periods.

FIG. 11 is a diagram illustrating an arrangement example of pixels withdifferent exposure periods.

FIG. 12 is a diagram illustrating an arrangement example of pixels withdifferent exposure periods.

FIG. 13 is a diagram illustrating different exposure periods.

FIG. 14 is a diagram for explaining a configuration of an imageprocessing unit.

FIG. 15 is a diagram for explaining a configuration of a sensitivityclassified interpolating unit.

FIG. 16 is a diagram for explaining a configuration of an HDRsynthesizing unit.

FIG. 17 is a diagram for explaining a configuration of a flickercorrecting unit.

FIG. 18 is a diagram for explaining a configuration of a flickercorrecting unit.

FIG. 19 is a diagram for explaining a flicker ratio.

FIG. 20 is a diagram for explaining a configuration of an estimationoperating unit.

FIG. 21 is a diagram for explaining another configuration of theestimation operating unit.

FIG. 22 is a diagram for explaining a shutter function.

FIG. 23 is a diagram for explaining another configuration of the flickerestimating unit.

FIG. 24 is a diagram for explaining calculation of a flicker componentin frequency space.

FIG. 25 is a diagram for explaining a recording medium.

DESCRIPTION OF EMBODIMENTS

Embodiments for implementing the present technology (hereinafter,referred to as embodiments) will be described below. It should be notedthat description will be provided in the following order.

1. Flicker Occurrence Principle and Correction Principle 2.Configuration of Imaging Apparatus 3. Configuration of Image ProcessingUnit 4. Configuration of Flicker Correcting Unit 5. Calculation ofFlicker Ratio 6. First Embodiment of Flicker Suppression 7. SecondEmbodiment of Flicker Suppression 8. Third Embodiment of FlickerSuppression 9. Recording Medium <Flicker Occurrence Principle andCorrection Principle>

First, flicker occurrence principle and correction principle will bedescribed with reference to FIG. 1. Part A in FIG. 1 illustratestemporal change of illumination brightness under an environment where animage is picked up using a camera. Typically, because a commercial powersupply is an AC power supply of 50 Hz or 60 Hz, illumination light suchas light from a fluorescent light is likely to fluctuate at a frequencyof 100 Hz or 120 Hz.

It should be noted that while description is provided here using flickeras an example, the present technology described below can be alsoapplied to noise, or the like, occurring at a predetermined frequencylike flicker.

Graph A in FIG. 1 indicates time t on a horizontal axis and illuminationbrightness f(t) at each time t on a vertical axis. The illuminationlight brightness f(t) at time t can be expressed as follows when theillumination light brightness f(t) is decomposed into an average valuef_(D) of illumination light brightness and fluctuation f_(A)(t) from theaverage value of the illumination light brightness.

f(t)=f _(D) +f _(A)(t)  (1)

The average value f_(D) of the illumination light brightness is aconstant value regardless of time t, and fluctuation f_(A)(t) from theaverage value becomes a value periodically fluctuating according to afrequency of illumination. Further, when the brightness of theillumination light is a cycle of f(t) is set to T, the followingrelationship holds.

[Math. 3]

f(t+T)=f(t)

∫_(t) ^(t+T) f(τ)dτ=f _(D)

∫_(t) ^(t+T) f _(A)(τ)dτ=0  (2)

Flicker correction processing is processing for removing influence offluctuation f_(A)(t) from the average value of the illumination lightbrightness, from an observation image, that is, an image picked up usinga camera.

Part B in FIG. 1 illustrates a pattern diagram of an exposure timing ofan imaging device in which an imaging timing is different for each rowas in a CMOS image sensor. Part B indicates time t on a horizontal axisand row y of the imaging device on a vertical axis. The exampleillustrated in the diagram is an example in the case where images ofcontinuous image frames are picked up at regular intervals S, andillustrates exposure timings when two images of a frame 1 and a frame 2are picked up. When each frame image is picked up, exposure issequentially executed from an upper row from a lower row of the imagingdevice.

Because an exposure timing when each frame image is picked up isdifferent for each row of the imaging device, influence of accumulatedillumination light is also different for each row. For example, exposurecompletion time of a predetermined pixel of the imaging device in anexposure period E is set at t. When a sum of illumination light whilethe pixel is exposed under conditions in which there is influence offlicker is set at F_(A)(t, E), F_(A)(t, E) can be expressed as follows.

[Math. 4]

F _(A)(t,E)=∫_(t-E) ^(t) f(τ)dτ=f _(D) ·E+∫ _(t-E) ^(t) f _(A)(τ)dτ  (3)

A sum of illumination light under ideal conditions in which there is noflicker is set at F_(D)(t, E). Because F_(D)(t, E) is not affected byflicker, fluctuation from the average value of the illumination lightbrightness becomes f_(A)(t)=0, and F_(D)(t, E) can be expressed asfollows.

F _(D)(t,E)=f _(D) ×E  (4)

Here a “flicker component” is defined as a ratio between an ideal imagein which there is no flicker and an image which is affected by flicker.The flicker component is equal to a ratio of a total amount ofillumination light while pixels accumulate the illumination light.Therefore, a flicker component g(t, E) of a pixel at exposure completiontime t in the imaging device in the exposure period E can be formulatedas expressed in the following formula (5).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 5} \right\rbrack & \; \\{{g_{t}\left( {t,E} \right)} = {\frac{F_{A}\left( {t,E} \right)}{F_{D}\left( {t,E} \right)} = \frac{{Ef}_{D} + {\int_{t - E}^{t}{{f_{A}(\tau)}{\tau}}}}{{Ef}_{D}}}} & (5)\end{matrix}$

Part C in FIG. 1 indicates an exposure completion timing t of each pixelof an image on a horizontal axis and a flicker component g(t, E) on avertical axis, and schematically illustrates relationship between theexposure completion timing t and the flicker component g(t, E). Itshould be noted that, as described above, because the illumination lightfluctuates, the flicker component also has periodicity. Therefore, ifthe flicker component g(t, E) can be obtained once, it is basicallypossible to estimate the flicker component g(t, E) corresponding to anyexposure completion timing t.

It should be noted that the exposure completion timing as illustrated inpart B in FIG. 1 changes in units of row of the imaging device.Accordingly, as illustrated in part C in FIG. 1, the flicker componentg(t, E) becomes a value different according to the exposure completiontiming T of each row.

Part A in FIG. 2 is a pattern diagram of influence of flicker occurringin an output image of the imaging device which is affected by flicker.Because the exposure completion timing is different for each row, abright and dark stripe pattern in units of row appears in the outputimage.

Part B in FIG. 2 is a graph g(t0, y, E) of a flicker component in eachrow of the output image. t0 indicates time at which exposure in thefirst row is finished, and y indicates a target row. A data processingunit of the imaging apparatus (camera) can calculate a flicker componentg(t, E) corresponding to t from graph C in FIG. 1 based on the exposureperiod E when an image is picked up and the exposure completion timing tof each row y.

Specifically, a unit of a period from exposure of a predetermined rowbeing finished until exposure of the next row below is finished isdefined as 1 [line]. When the unit is defined in this manner, g(t0, y,E) and g(t, E) can be converted as follows.

g _(y)(t,y,E)=gt(t+y,E)  (6)

A data processing unit of the imaging apparatus (camera) can calculate aflicker component g(t, E) corresponding to t in graph C in FIG. 1 basedon the exposure period E when an image is picked up and the exposurecompletion timing t of each row y. For example, when the exposurecompletion time in the a-th row illustrated in FIG. 2 is set at t, aflicker component g(t, E) corresponding to t can be calculated fromgraph C in FIG. 1. If the flicker component g(t,E) of a pixel at theexposure completion time t in the imaging device in the exposure periodE can be known, it is possible to estimate a flicker component g(y) ofeach row of the imaging device.

FIG. 3 illustrates flicker correction principle. FIG. 3 includes thefollowing diagrams.

Part A in FIG. 3: image including a flicker component (=part A in FIG.2)Part B in FIG. 3: flicker correction function (=reciprocal of part B inFIG. 2)Part C in FIG. 3: flicker correction image (=part A in FIG. 3×part B inFIG. 3)

For example, it is possible to obtain an ideal image which is notaffected by flicker illustrated in part C in FIG. 3 by measuring aflicker component g(y) of each row using the above-described method andmultiplying each pixel value of the observation image illustrated inpart A in FIG. 3, that is, an image picked up using the camera by thereciprocal of the flicker component g(y) illustrated in part B in FIG.3.

<Configuration of Imaging Apparatus>

An image processing apparatus to which the present technology is appliedreceives input of a plurality of picked up images for which differentexposure periods are set and generates and outputs corrected images fromwhich flicker components are removed or reduced to generate, forexample, a high dynamic range image. The image processing apparatus towhich the present technology is applied, for example, synthesizes aplurality of picked up images for which different exposure periods areset to generate and output a high dynamic range image in which moreaccurate pixel values are set from a low brightness portion to a highbrightness portion.

In the image processing apparatus to which the present technology isapplied, processing of calculating a flicker component for each of aplurality of images for which different exposure periods are set is notexecuted. Processing of calculating a flicker component is executed ononly a picked up image in one exposure period, and processing ofestimating flicker components included in picked up images for which theother different exposure periods are set is executed by utilizing theflicker component calculated based on the picked up image of the oneexposure period. Such image processing apparatus will be described.

FIG. 4 is a diagram illustrating a configuration of an embodiment of theimage processing apparatus to which the present technology is applied.Here, description will be provided using an example of an imagingapparatus including an image processing apparatus.

An imaging apparatus 100 illustrated in FIG. 4 is configured to includean optical lens 101, an imaging device 102, an image processing unit103, a signal processing unit 104 and a control unit 105. In the imagingapparatus 100 illustrated in FIG. 4, light incident through the opticallens 101 is incident on an imaging unit, for example, the imaging device102 configured with a CMOS image sensor, or the like, and image dataobtained through photoelectric conversion is outputted. The output imagedata is inputted to the image processing unit 103.

The output image of the imaging device 102 is a so-called mosaic imagein which any pixel value of R, G and B is set at each pixel. The imageprocessing unit 103 performs the above-described flicker correctionprocessing, and, further, processing of generating a high dynamic range(HDR) image based on processing of synthesizing a long-period exposureimage and a short-period exposure image, or the like.

The output of the image processing unit 103 is inputted to the signalprocessing unit 104. The signal processing unit 104 executes signalprocessing which is performed in a typical camera, such as, for example,white balance (WB) adjustment and gamma correction, to generate anoutput image 120. The output image 120 is stored in a storage unit whichis not illustrated or outputted to a display unit.

The control unit 105 outputs a control signal to each unit according toa program stored in, for example, a memory which is not illustrated, tocontrol various kinds of processing.

It should be noted that while description will be continued here whilethe imaging device 102, the image processing unit 103, the signalprocessing unit 104 and the control unit 105 are respectivelyillustrated as separate blocks, all or part of these units may beintegrally configured.

For example, it is also possible to integrally configure the imagingdevice 102, the image processing unit 103, the signal processing unit104 and the control unit 105 as a laminate structure. Further, it isalso possible to integrally configure the imaging device 102, the imageprocessing unit 103 and the signal processing unit as a laminatestructure. Still further, it is also possible to integrally configurethe imaging device 102 and the image processing unit 103 as a laminatestructure.

Further, the configuration of the imaging apparatus 100 is not limitedto the configuration illustrated in FIG. 4 and may be otherconfigurations. For example, it is also possible to divide the imageprocessing unit 103 into a plurality of image processing units andconfigure a laminate structure by integrating part of the imageprocessing units and the imaging device 102.

Next, an example of an exposure control configuration of the imagingdevice 102 will be described with reference to FIG. 5. In the imagingapparatus 100, a long-period exposure pixel and a short-period exposurepixel are set in units of pixels included in one picked up image, and ahigh dynamic range image is generated through synthesis processing(α-blend) between these pixels. This exposure period control isperformed through control by the control unit 105.

FIG. 5 is a diagram illustrating an example of exposure period settingof the imaging device 102. As illustrated in FIG. 5, pixels composingthe imaging device are sorted into two types of pixels of pixels set tofirst exposure conditions (short-period exposure) and pixels set tosecond exposure conditions (long-period exposure).

In FIG. 5, pixels which are shaded are images exposed under the firstexposure conditions, while pixels which are not shaded are pixelsexposed under the second exposure conditions. As in FIG. 5, a pixelarray which has pixels exposed for different exposure periods likeshort-period exposure pixels and long-period exposure pixels within oneimaging device is referred to as a spatially varying exposure (SVE)array.

The pixel arrangement illustrated in FIG. 5 is arrangement of R pixels,G pixels and B pixels arranged in one to eight rows and one to eightcolumns. FIG. 5 illustrates part of the image sensor, and R pixels, Gpixels and B pixels arranged in other rows and columns other than one toeight rows and one to eight columns have the same configurations asthose of the R pixels, the G pixels and the B pixels arranged in one toeight rows and one to eight columns.

In the following description, for example, while a pixel is described as10(m, n), m indicates a row and n indicates a column. Further, the rowis a horizontal direction in which a horizontal signal line (notillustrated) is disposed, while the column is a vertical direction inwhich a vertical signal line (not illustrated) is disposed. For example,a pixel 200(2, 1) indicates a pixel positioned in the second row in thefirst column. Further, here, an upper left pixel is set as a pixel200(1, 1), and a position of each pixel is indicated based on this pixel200(1, 1). Other drawings will be indicated in the same way.

A configuration of the image sensor in a horizontal direction (ahorizontal direction in FIG. 5 and a row direction) will be described.In the first row, an R pixel 200(1, 1), a G pixel 200(1, 2), a G pixel200(1, 4), an R pixel 200(1, 5), a G pixel 200(1, 6) and a G pixel200(1, 8) exposed under the first exposure conditions and an R pixel200(1, 3) and an R pixel 200(1, 7) exposed under the second exposureconditions are arranged.

In this case, R pixels and G pixels are alternately arranged in thefirst row. Further, as the R pixels 200 in the first row, pixels exposedunder the first exposure conditions and pixels exposed under the secondexposure conditions are alternately arranged. Still further, the Gpixels 200 in the first row are all exposed under the first exposureconditions.

In the second row, a B pixel 200(2, 2) and a B pixel 200(2, 6) exposedunder the first exposure conditions, and a G pixel 200(2, 1), a G pixel200(2, 3), a B pixel 200(2, 4), a G pixel 200(2, 5), a G pixel 200(2, 7)and a B pixel 200(2, 8) exposed under the second exposure conditions arearranged.

In this case, in the second row, the G pixels and the B pixels arealternately arranged. Further, as the B pixels 200 in the second row,pixels exposed under the first exposure conditions and pixels exposedunder the second exposure conditions are alternately arranged. Stillfurther, the G pixels 200 in the second row are all exposed under thesecond exposure conditions.

While the third row is different from the first row in that pixels arearranged, starting from an R pixel (3, 1) exposed under the secondexposure conditions, as in the first row, R pixels and G pixels arealternately arranged, as the arranged R pixels 200, pixels exposed underthe first exposure conditions and pixels exposed under the secondexposure conditions are alternately arranged, and the arranged G pixels200 are all exposed under the first exposure conditions.

While the fourth row is different from the second row in that pixels arearranged, starting from a G pixel (4, 1) and a B pixel 200(4, 2) exposedunder the second exposure conditions, as in the second row, the G pixelsand the B pixels are alternately arranged, as the arranged B pixels,pixels exposed under the first exposure conditions and pixels exposedunder the second exposure conditions are alternately arranged, and thearranged G pixels 200 are all exposed under the second exposureconditions.

R pixels, G pixels and B pixels are respectively arranged in the fifthrow in a similar manner to in the first row, in the sixth row in asimilar manner to in the second row, in the seventh row in a similarmanner to in the third row, and in the eighth row in a similar manner toin the fourth row.

While description will be provided using an example of the pixelarrangement illustrated in FIG. 5 in the following description, thepresent technology is not limited to be applied to the pixel arrangementillustrated in FIG. 5, and can be also applied to other pixelarrangement. Examples of other pixel arrangement will be described withreference to FIG. 6 to FIG. 12.

FIG. 6 is a diagram illustrating another example of the pixelarrangement. In the first row in the pixel arrangement illustrated inFIG. 6, an R pixel 210(1, 1), a G pixel 210(1, 2), an R pixel 210(1, 3),a G pixel 210(1, 4), an R pixel 210(1, 5), a G pixel 210(1, 6), an Rpixel 210(1, 7) and a G pixel 210(1, 8) exposed under the first exposureconditions are arranged.

In this case, in the first row, R pixels and G pixels, which are allexposed under the first exposure conditions (short-period exposure), arealternately arranged.

In the second row, a G pixel 210(2, 1), a B pixel 210(2, 2), a G pixel210(2, 3), a B pixel 210(2, 4), a G pixel 210(2, 5), a B pixel 210(2,6), a G pixel 210(2, 7) and a B pixel 210(2, 8) exposed under the firstexposure conditions are arranged.

In this case, in the second row, G pixels and B pixels, which are allexposed under the first exposure conditions (short-period exposure), arealternately arranged.

In the third row, an R pixel 210(3, 1), a G pixel 210(3, 2), an R pixel210(3, 3), a G pixel 210(3, 4), an R pixel 210(3, 5), a G pixel 210(3,6), an R pixel 210(3, 7) and a G pixel 210(3, 8) exposed under thesecond exposure conditions are arranged.

In this case, in the third row, R pixels and G pixels, which are allexposed under the second exposure conditions (long-period exposure), arealternately arranged.

In the fourth row, a G pixel 210(4, 1), a B pixel 210(4, 2), a G pixel210(4, 3), a B pixel 210(4, 4), a G pixel 210(4, 5), a B pixel 210(4,6), a G pixel 210(4, 7) and a B pixel 210(4, 8) exposed under the secondexposure conditions are arranged.

In this case, in the fourth row, G pixels and B pixels, which are allexposed under the second exposure conditions (long-period exposure), arealternately arranged.

R pixels, G pixels and B pixels are respectively arranged in the fifthrow in a similar manner to in the first row, in the sixth row in asimilar manner to in the second row, in the seventh row in a similarmanner to in the third row, and in the eighth row in a similar manner toin the fourth row.

The present technology can be also applied to such pixel arrangement.

FIG. 7 is a diagram illustrating another example of the pixelarrangement. In the first row in the pixel arrangement illustrated inFIG. 7, an R pixel 220(1, 1), a G pixel 220(1, 2), an R pixel 220(1, 5)and a G pixel 220(1, 6) exposed under the first exposure conditions andan R pixel 220(1, 3), a G pixel 220(1, 4), an R pixel 220(1, 7) and a Gpixel 220(1, 8) exposed under the second exposure conditions arearranged.

In this case, in the first row, R pixels and G pixels are alternatelyarranged, and as each of the R pixels and the G pixels, pixels exposedunder the first exposure conditions and pixels exposed under the secondexposure conditions are alternately arranged.

In the second row, a G pixel 220(2, 1), a B pixel 220(2, 2), a G pixel220(2, 5) and a B pixel 220(2, 6) exposed under the first exposureconditions and a G pixel 220(2, 3), a B pixel 220(2, 4), a G pixel220(2, 7) and a B pixel 220(2, 8) exposed under the second exposureconditions are arranged.

In this case, in the second row, G pixels and B pixels are alternatelyarranged, and as each of the G pixels and the B pixels, pixels exposedunder the first exposure conditions and pixels exposed under the secondexposure conditions are alternately arranged.

While the third row is different from the first row in that pixels arearranged, starting from an R pixel 220(3,1) and a G pixel 220(3, 2)exposed under the second exposure conditions, as in the first row, Rpixels and G pixels are alternately arranged, and as each of thearranged R pixels and G pixels, pixels exposed under the first exposureconditions and pixels exposed under the second exposure conditions arealternately arranged.

While the fourth row is different from the second row in that pixels arearranged, starting from a G pixel 220(4, 1) and a B pixel 220(4, 2)exposed under the second exposure conditions, as in the second row, thearranged G pixels and B pixels are alternately arranged, and as each ofthe G pixels and the B pixels, pixels exposed under the first exposureconditions and pixels exposed under the second exposure conditions arealternately arranged.

R pixels, G pixels and B pixels are respectively arranged in the fifthrow in a similar manner to in the first row, in the sixth row in asimilar manner to in the second row, in the seventh row in a similarmanner to in the third row, and in the eighth row in a similar manner toin the fourth row.

The present technology can be also applied to such pixel arrangement.

FIG. 8 is a diagram illustrating another example of the pixelarrangement. In the pixel arrangement illustrated in FIG. 8, four pixelsof 2×2 vertically and horizontally are illustrated with the same color,and pixels of the first exposure conditions and pixels of the secondexposure conditions are arranged in a checkered pattern.

Among four pixels of 2×2 arranged in the first row and the second row,four pixels of an R pixel 230(1, 1), an R pixel 230(1, 2), an R pixel230(2, 1) and an R pixel 230(2, 2) are R (red) pixels, the R pixel230(1, 1) and the R pixel 230(2, 2) being exposed under the secondexposure conditions, and the R pixel 230(1, 2) and the R pixel 230(2, 1)being exposed under the first exposure conditions. Four red pixelshaving such arrangement will be described as an R pixel block.

Among four pixels of 2×2 arranged in the first row and the second rowadjacent to such an R pixel block, four pixels of a G pixel 230(1, 3), aG pixel 230(1, 4), a G pixel 230(2, 3) and a G pixel 230(2, 4) are G(green) pixels, the G pixel 230(1, 3) and the G pixel 230(2, 4) beingexposed under the second exposure conditions, and the G pixel 230(1, 4)and the G pixel 230(2, 3) being exposed under the first exposureconditions. Four green pixels having such arrangement will be describedas a G pixel block.

In the first row and the second row, R pixel blocks and G pixel blocksare alternately arranged.

In the third row and the fourth row, G pixel blocks each constitutedwith a G pixel 230(3, 1), a G pixel 230(3, 2), a G pixel 230(4, 1) and aG pixel 230(4, 2) are arranged.

Among four pixels of 2×2 arranged in the third row and the fourth rowadjacent to the G pixel block, four pixels of a B pixel 230(3, 3), a Bpixel 230(3, 4), a B pixel 230(4, 3) and a B pixel 230(4, 4) are B(green) pixels, the B pixel 230(3, 3) and the B pixel 230(4, 4) beingexposed under the second exposure conditions, and the B pixel 230(3, 4)and the B pixel 230(4, 3) being exposed under the first exposureconditions. Four blue pixels having such arrangement will be describedas a B pixel block.

In the third row and the fourth row, G pixel blocks and B pixel blocksare alternately arranged.

In the fifth row and the sixth row, as in the first row and the secondrow, R pixel blocks and G pixel blocks are alternately arranged. In theseventh row and the eighth row, as in the third row and the fourth row,G pixel blocks and B pixel blocks are alternately arranged.

The present technology can be also applied to such pixel arrangement.

FIG. 9 is a diagram illustrating another example of the pixelarrangement. While the pixel arrangement illustrated in FIG. 9 has thesame color arrangement as the pixel arrangement illustrated in FIG. 8,the pixel arrangement illustrated in FIG. 9 is different from the pixelarrangement illustrated in FIG. 8 in arrangement of pixels havingdifferent exposure conditions.

Among four pixels of 2×2 arranged in the first row and the second row,among four pixels of an R′ pixel block constituted with an R pixel240(1, 1), an R pixel 240(1, 2), an R pixel 240(2, 1) and an R pixel240(2, 2), the R pixel 240(1, 1) and the R pixel 240(1, 2) are exposedunder the first exposure conditions, and the R pixel 240(2, 1) and the Rpixel 240(2, 2) are exposed under the second exposure conditions.

Among four pixels of 2×2 arranged in the first row and the second rowadjacent to such an R′ pixel block, among four pixels of a G′ pixelblock constituted with a G pixel 240(1, 3), a G pixel 240(1, 4), a Gpixel 240(2, 3) and a G pixel 240(2, 4), the G pixel 240(1, 3) and the Gpixel 240(1, 4) are exposed under the first exposure conditions, and theG pixel 240(2, 3) and the G pixel 240(2, 4) are exposed under the secondexposure conditions.

In the third row and the fourth row, G′ pixel blocks each constitutedwith a G pixel 240(3, 1), a G pixel 240(3, 2), a G pixel 240(4, 1) and aG pixel 240(4, 2) are arranged.

Among four pixels of 2×2 arranged in the third row and the fourth rowadjacent to the G′ pixel block, among four pixels of a B′ pixel blockconstituted with a B pixel 240(3, 3), a B pixel 240(3, 4), a B pixel240(4, 3) and a B pixel 240(4, 4), the B pixel 240(3, 3) and the B pixel240(3, 4) are exposed under the first exposure conditions, and the Bpixel 240(4, 3) and the B pixel 240(4, 4) are exposed under the secondexposure conditions.

In the fifth row and the sixth row, as in the first row and the secondrow, R′ pixel blocks and G′ pixel blocks are alternately arranged. Inthe seventh row and the eighth row, as in the third row and the fourthrow, G′ pixel blocks and B′ pixel blocks are alternately arranged.

In the pixel arrangement illustrated in FIG. 9, pixels exposed under thefirst exposure conditions are arranged in odd rows, and pixels exposedunder the second exposure conditions are arranged in even rows.

The present technology can be also applied to such pixel arrangement.

FIG. 10 is a diagram illustrating another example of the pixelarrangement. While the pixel arrangement illustrated in FIG. 10 has thesame color arrangement as the pixel arrangement illustrated in FIG. 8,the pixel arrangement illustrated in FIG. 10 is different from the pixelarrangement illustrated in FIG. 8 in arrangement of pixels havingdifferent exposure conditions.

Among four pixels of 2×2 arranged in the first row and the second row,among four pixels of an R″ pixel block constituted with an R pixel250(1, 1), an R pixel 250(1, 2), an R pixel 250(2, 1) and an R pixel250(2, 2), the R pixel 250(1, 1) and the R pixel 250(2, 1) are exposedunder the first exposure conditions, and the R pixel 250(1, 2) and the Rpixel 250(2, 2) are exposed under the second exposure conditions.

Among four pixels of 2×2 arranged in the first row and the second rowadjacent to such an R″ pixel block, among four pixels of a G″ pixelblock constituted with a G pixel 250(1, 3), a G pixel 250(1, 4), a Gpixel 250(2, 3) and a G pixel 250(2, 4), the G pixel 250(1, 3) and the Gpixel 250(2, 3) are exposed under the first exposure conditions, and theG pixel 250(1, 4) and the G pixel 250(2, 4) are exposed under the secondexposure conditions.

In the third row and the fourth row, G″ pixel blocks each constitutedwith a G pixel 250(3, 1), a G pixel 250(3, 2), a G pixel 250(4, 1) and aG pixel 250(4, 2) are arranged.

Among four pixels of 2×2 arranged in the third row and the fourth rowadjacent to the G″ pixel block, among four pixels of a B″ pixel blockconstituted with a B pixel 250(3, 3), a B pixel 250(3, 4), a B pixel250(4, 3) and a B pixel 250(4, 4), the B pixel 250(3, 3) and the B pixel250(4, 3) are exposed under the first exposure conditions, and the Bpixel 250(3, 4) and the B pixel 250(4, 4) are exposed under the secondexposure conditions.

In the fifth row and the sixth row, as in the first row and the secondrow, R″ pixel blocks and G″ pixel blocks are alternately arranged. Inthe seventh row and the eighth row, as in the third row and the fourthrow, G″ pixel blocks and B″ pixel blocks are alternately arranged.

In the pixel arrangement illustrated in FIG. 10, pixels exposed underthe first exposure conditions are arranged in odd rows, and pixelsexposed under the second exposure conditions are arranged in even rows.

The present technology can be also applied to such pixel arrangement.

FIG. 11 is a diagram illustrating another example of the pixelarrangement. In the pixel arrangement illustrated in FIG. 11, in thefirst row, a G pixel 260(1, 1), an R pixel 260(1, 2), a G pixel 260(1,3), a B pixel 260(1, 4), a G pixel 260(1, 5), an R pixel 260(1, 6), a Gpixel 260(1, 7) and a B pixel 260(1, 8) exposed under the first exposureconditions are arranged.

In this case, in the first row, R pixels, G pixels and B pixels, whichare all exposed under the first exposure conditions (short-periodexposure), are arranged.

In the second row, a B pixel 260(2, 1), a G pixel 260(2, 2), an R pixel260(2, 3), a G pixel 260(2, 4), a B pixel 260(2, 5), a G pixel 260(2,6), an R pixel 260(2, 7) and a G pixel 260(2, 8) exposed under thesecond exposure conditions are arranged.

In this case, in the second row, R pixels, G pixels and B pixels, whichare all exposed under the second exposure conditions (long-periodexposure), are arranged.

In the third row, a G pixel 260(3, 1), a B pixel 260(3, 2), a G pixel260(3, 3), an R pixel 260(3, 4), a G pixel 260(3, 5), a B pixel 260(3,6), a G pixel 260(3, 7) and an R pixel 260(3, 8) exposed under the firstexposure conditions are arranged.

In this case, in the third row, R pixels, G pixels and B pixels, whichare all exposed under the first exposure conditions (short-periodexposure), are arranged.

In the fourth row, an R pixel 260(4, 1), a G pixel 260(4, 2), a B pixel260(4, 3), a G pixel 260(4, 4), an R pixel 260(4, 5), a G pixel 260(4,6), a B pixel 260(4, 7) and a G pixel 260(4, 8) exposed under the secondexposure conditions are arranged.

In this case, in the fourth row, R pixels, G pixels and B pixels, whichare all exposed under the second exposure conditions (long-periodexposure) are arranged.

R pixels, G pixels and B pixels are respectively arranged in the fifthrow in a similar manner to in the first row, in the sixth row in asimilar manner to in the second row, in the seventh row in a similarmanner to in the third row, and in the eighth row in a similar manner toin the fourth row.

The present technology can be also applied to such pixel arrangement.

As described above, the present technology can be applied to an imagingapparatus including, for example, a charge coupled device (CCD) sensor,a complementary metal oxide semiconductor (CMOS) sensor, or the like, asan imaging device included in the imaging apparatus. Further, thepresent technology can be applied to an image sensor in which threepixels which output color light of R (red), G (green) and B (blue) arearranged. Still further, the present technology can be also applied toan image sensor in which four pixels which output color light of R(red), G (green), B (blue) and W (white) are arranged as illustrated inFIG. 12.

Four pixels which output color light of R (red), G (green), B (blue) andW (white) are, for example, arranged in matrix in a display area asillustrated in FIG. 12. A W pixel functions as a pixel havingpanchromatic spectral sensitivity, and an R pixel, a G pixel and a Bpixel function as pixels having spectral sensitivity havingcharacteristics of respective colors.

FIG. 12 is a diagram illustrating another example of the pixelarrangement and illustrates an example of the pixel arrangementincluding W pixels. In the pixel arrangement illustrated in FIG. 12, inthe first row, a G pixel 270(1, 1), an R pixel 270(1, 2), a W pixel270(1, 3), a B pixel 270(1, 4), a G pixel 270(1, 5), an R pixel 270(1,6), a W pixel 270(1, 7) and a B pixel 270(1, 8) exposed under the firstexposure conditions are arranged.

In this case, in the first row, R pixels, G pixels, B pixels and Wpixels, which are all exposed under the first exposure conditions(short-period exposure) are arranged.

In the second row, an R pixel 270(2, 1), a W pixel 270(2, 2), a B pixel270(2, 3), a G pixel 270(2, 4), an R pixel 270(2, 5), a W pixel 270(2,6), a B pixel 270(2, 7) and a G pixel 270(2, 8) exposed under the secondexposure conditions are arranged.

In this case, in the second row, R pixels, G pixels, B pixels and Wpixels, which are all exposed under the second exposure conditions(long-period exposure), are arranged.

In the third row, a W pixel 270(3, 1), a B pixel 270(3, 2), a G pixel270(3, 3), an R pixel 270(3, 4), a W pixel 270(3, 5), a B pixel 270(3,6), a G pixel 270(3, 7) and an R pixel 270(3, 8) exposed under the firstexposure conditions are arranged.

In this case, in the third row, R pixels, G pixels, B pixels and Wpixels, which are all exposed under the first exposure conditions(short-period exposure), are arranged.

In the fourth row, a B pixel 270(4, 1), a G pixel 270(4, 2), an R pixel270(4, 3), a W pixel 270(4, 4), a B pixel 270(4, 5), a G pixel 270(4,6), an R pixel 270(4, 7) and a W pixel 270(4, 8) exposed under thesecond exposure conditions are arranged.

In this case, in the fourth row, R pixels, G pixels, B pixels and Wpixels, which are all exposed under the second exposure conditions(long-period exposure), are arranged.

R pixels, G pixels, B pixels and W pixels are respectively arranged inthe fifth row in a similar manner to in the first row, in the sixth rowin a similar manner to in the second row, in the seventh row in asimilar manner to in the third row, and in the eighth row in a similarmanner to in the fourth row.

The present technology can be also applied to such pixel arrangement.

The pixel arrangement described with reference to FIG. 5 to FIG. 12 isan example, and the present technology can be also applied to pixelarrangement which is not described.

Further, when one image is picked up, while description will becontinued using the example as described above where an image is pickedup using short-period exposure (the first exposure conditions) andlong-period exposure (the second exposure conditions) at the same time,the present technology can be also applied to a case where an image uponshort-period exposure and an image upon long-period exposure areacquired by alternately picking up a short-period exposure image and along-period exposure image with normal pixels without separating pixelsfor short-period exposure from pixels for long-period exposure.

In this case, because imaging timings are different, the presenttechnology can be applied by using a matrix which takes into account theimaging timings as a matrix to be used for operation upon flickercorrection which will be described later.

Further, while, in the above-described example, description has beenprovided using an example of an imaging apparatus which picks up imagesusing two types of exposure periods of short-period exposure andlong-period exposure, the present technology can be also applied to animaging apparatus in which picked up images with three or more types ofexposure periods are combined.

When picked up images with three or more types of exposure periods arecombined, it is also possible to estimate a first flicker component froma first exposure image and a second exposure image and convert andestimate a third flicker component from the first flicker component.Further, it is also possible to obtain solution by generating a matrixby combining all the first exposure image, the second exposure image andthe third exposure image. A method for obtaining flicker components willbe described later.

Further, while, in the above-described embodiment, an example in thecase where spectral sensitivity of pixels of the imaging device is RGBor RG+W has been described, the spectral sensitivity does not become aconstraint when the present technology is used. That is, it is alsopossible to use pixels having spectral sensitivity other than RGB andRGB+W. For example, it is also possible to combine four rows including Gin addition to complementary colors such as Y (yellow), C (cyan) and M(magenta).

In the following description, description will be provided using anexample of the pixel arrangement illustrated in FIG. 5.

FIG. 13 illustrates a setting example of exposure periods of respectivepixels. Pixels set at the first exposure conditions (short-periodexposure) are subjected to exposure processing of a short period. Pixelsset at the second exposure conditions (long-period exposure) aresubjected to exposure processing of a long period. It should be notedthat this exposure control in units of pixels is performed by, forexample, a control unit 105 of the imaging apparatus 100 illustrated inFIG. 4 outputting a control signal to the imaging device 102.

<Configuration of Image Processing Unit>

Next, details of the image processing unit 103 of the imaging apparatus100 illustrated in FIG. 4 will be described. First, processing executedby the image processing unit 103 will be described with reference toFIG. 14. As illustrated in FIG. 14, the image processing unit 103 has asensitivity classified interpolating unit 311, a flicker correcting unit312 and an HDR synthesizing unit (high dynamic range image synthesizingunit) 313.

The sensitivity classified interpolating unit 311 receives input of animage of an SVE array having short-period exposure pixels andlong-period exposure pixels within one imaging device as illustrated inFIG. 5, and generates and outputs a first exposure image 141 in whichthe whole screen is exposed for a short period and a second exposureimage 142 in which the whole screen is exposed for a long period. Acolor array of the outputted image may be equal to a color array of theinputted image (in the present example, a Bayer array), or may be animage after being subjected to demosaicing in which RGB are put togetherin one pixel position. Here, an example will be described where thecolor array of the outputted image is equal to the color array of theinputted image (in the present example, a Bayer array).

FIG. 15 illustrates a detailed configuration example of the sensitivityclassified interpolating unit 311. As illustrated in FIG. 11, thesensitivity classified interpolating unit 311 has extracting units 331and 332 configured to extract only pixels of any sensitivity ofshort-period exposure pixels and long-period exposure pixels, andinterpolation processing units 333 and 334 configured to set pixelvalues of pixel portions of other sensitivity by utilizing pixels havingrespective sensitivity and generate a first exposure image 141 formedwith only low-sensitivity pixels (short-period exposure pixels) and asecond exposure image 142 formed with only high-sensitivity pixels(long-period exposure pixels).

The extracting units 331 and 332 extract pixels of sensitivity and colordesired to be interpolated from peripheral pixels, and the interpolationprocessing units 333 and 334 perform interpolation processing. It shouldbe noted that it is also possible to perform interpolation by utilizinga method which uses a simple LPF for a pixel value of sensitivityaccording to the generated image, a method in which an edge direction ofan image is estimated from peripheral pixels and interpolation isperformed using a pixel value in a direction along the edge as areference pixel value, or the like.

The sensitivity classified interpolating unit 311 receives input of anSVE array having long-period exposure pixels and short-period exposurepixels within the imaging device as illustrated in FIG. 4 by applyingthe configuration in FIG. 15, and generates and outputs a first exposureimage 141 in which the whole screen is exposed for a short period and asecond exposure image 142 in which the whole screen is exposed for along period.

Description will be returned to description of the image processing unit103 illustrated in FIG. 14. The first exposure image 141 and the secondexposure image 142 outputted from the sensitivity classifiedinterpolating unit 311 are supplied to the flicker correcting unit 312.The flicker correcting unit 312 generates a flicker-corrected firstexposure image 143 and a flicker-corrected second exposure image 144 inwhich flicker components are suppressed and supplies theflicker-corrected first exposure image 143 and the flicker-correctedsecond exposure image 144 to the HDR synthesizing unit 313.

Details of a configuration and operation of the flicker correcting unit312 will be described later as the first to the third embodiments, anddescription of a configuration and operation of the image processingunit 103 illustrated in FIG. 14 will be continued.

The HDR synthesizing unit (high dynamic range image synthesizing unit)313 has a configuration as illustrated in FIG. 16 and performs HDRsynthesis. The HDR synthesizing unit 313 illustrated in FIG. 16 hasexposure correcting units 351 and 352, a blend coefficient calculatingunit 353 and a blend processing unit 354.

The exposure correcting units 351 and 352 match brightness ofcorresponding pixels of the flicker-corrected first exposure image 143and the flicker-corrected second exposure image 144 by multiplying aconstant according to an exposure period. For example, when the exposureratio is 1:2, a pixel value of the flicker-corrected first exposureimage 143 which is a short-period exposure image is multiplied by 2,while a pixel value of the flicker-corrected second exposure image 144which is a long-period exposure image is multiplied by 1.

The blend coefficient calculating unit 353 calculates a blendcoefficient which indicates how much ratio in units of correspondingpixels should be used to blend a pixel value of the flicker-correctedfirst exposure image 143 which is a short-period exposure image afterexposure is corrected and a pixel value of the flicker-corrected secondexposure image 144 which is a long-period exposure image after exposureis corrected.

In this blend coefficient calculation, a blend coefficient is set suchthat, for example, weight of a pixel value of a short-period exposureimage is set greater in a high brightness area, while weight of a pixelvalue of a long-period exposure image is set greater in a low brightnessarea. Through such coefficient setting processing, it is possible toexpress pixel values from the low brightness area to the high brightnessarea with higher precision.

The blend processing unit 354 executes processing of blending thecorresponding pixel values of the flicker-corrected first exposure image143 for which exposure is corrected and the flicker-corrected secondexposure image 144 for which exposure is corrected according to theblend coefficient calculated by the blend coefficient calculating unit353 and sets each pixel value of a flicker-corrected HDR image 145.

It should be noted that when a pixel value of the flicker-correctedfirst exposure image 143 for which exposure is corrected is S, a pixelvalue of the flicker-corrected second exposure image 144 for whichexposure is corrected is L, and a blend coefficient is a where 0≦α≦1, apixel value H of the flicker-corrected HDR image 231 can be calculatedusing the following formula.

H=(1−α)×S+α×L

The HDR synthesizing unit (high dynamic range image synthesizing unit)313 generates and outputs the flicker-corrected HDR image 145 in whichpixels values from the low brightness area to the high brightness areaare expressed with higher precision through these processing.

An HDR image is generated through processing of synthesizing images withdifferent exposure periods at the HDR synthesizing unit (high dynamicrange image synthesizing unit) 313. That is, blend processing isexecuted while, for example, weight of a pixel value of a short-periodexposure image is set larger in the high brightness area, while weightof a pixel value of a long-period exposure image is set larger in thelow brightness area, and a high dynamic range (HDR) image in which pixelvalues from the low brightness area to the high brightness area areexpressed with higher precision is generated and outputted.

As will be described later, in the image processing apparatus of thepresent disclosure, in a configuration in which a plurality of imageswith different exposure periods are inputted to generate a high dynamicrange (HDR) image, because a configuration is employed where only aflicker component corresponding to one reference exposure image iscalculated, and flicker components of images with other exposure periodsare estimated and calculated according to the flicker component of thereference exposure image, it is not necessary to perform processing ofindividually calculating flicker components of respective imagesaccording to respective exposure periods, so that it is possible torealize efficient processing.

It should be noted that the sensitivity classified interpolating unit311 illustrated in FIG. 15 and the HDR synthesizing unit 313 illustratedin FIG. 16 are examples, and it is also possible to employ otherconfigurations depending on arrangement of pixels under the firstexposure conditions and pixels under the second exposure conditions, forexample, the pixel arrangement as illustrated in FIG. 5. Further, thepresent technology can be applied even when the sensitivity classifiedinterpolating unit 311 and the HDR synthesizing unit 313 which performprocessing before and after processing at the flicker correcting unit312 have other configurations depending on processing at the flickercorrecting unit 312 which will be described later.

<Configuration of Flicker Correcting Unit>

FIG. 17 is a diagram illustrating an internal configuration example ofthe flicker correcting unit 312. The flicker correcting unit 312 has aplurality of embodiments according to operation relating to correction,and, while description will be provided below using examples of thefirst to the third embodiments, a configuration common among the firstto the third embodiments will be described with reference to FIG. 17.

Further, a flicker component for each image is estimated by, in thefirst and the second embodiments, performing operation using a ratiocalled a flicker ratio, and, in the third embodiment, performingoperation without using a ratio. Further, the first to the thirdembodiments are common in that a noise component for each image isestimated by performing operation which utilizes mutual relationshipbetween noises having periodicity, such as flicker, included in imagespicked up under different exposure conditions.

The flicker correcting unit 312 illustrated in FIG. 17 is configured toinclude a flicker estimating unit 371, a first exposure image flickercorrecting unit 372 and a second exposure image flicker correcting unit373. The flicker estimating unit 371 has a configuration as illustratedin FIG. 18, and generates a first flicker component 381 and a secondflicker component 382 from the inputted first exposure image 141 and thesecond exposure image 142.

The first flicker component 381 generated by the flicker estimating unit371 is supplied to the first exposure image flicker correcting unit 372.The first exposure image 141 is also supplied to the first exposureimage flicker correcting unit 372. The first exposure image flickercorrecting unit 372 performs flicker correction by multiplying the firstexposure image 141 by the reciprocal of the estimated first flickercomponent 381 for each row, and outputs a flicker-corrected firstexposure image 143.

In a similar manner, the second flicker component 382 generated by theflicker estimating unit 371 is supplied to the second exposure imageflicker correcting unit 373. The second exposure image 142 is alsosupplied to the second exposure image flicker correcting unit 373. Thesecond exposure image flicker correcting unit 373 performs flickercorrection by multiplying the second exposure image 142 by thereciprocal of the estimated second flicker component 382 for each row,and outputs a flicker-corrected second exposure image 144.

The flicker-corrected first exposure image 143 and flicker-correctedsecond exposure image 144 corrected in this manner are outputted to theHDR synthesizing unit 313 (FIG. 14).

FIG. 18 is a diagram illustrating an internal configuration example ofthe flicker estimating unit 371 illustrated in FIG. 17. The flickerestimating unit 371 illustrated in FIG. 18 is configured to include anintegrated value calculating unit 401, an integrated value calculatingunit 402, a dividing unit 403 and an estimation operating unit 404.

The first exposure image 141 is supplied to the integrated valuecalculating unit 401, and the second exposure image 142 is supplied tothe integrated value calculating unit 402. The integrated valuecalculating unit 401 integrates the first exposure image 141 in ahorizontal direction and outputs the integrated value to the dividingunit 403. In a similar manner, the integrated value calculating unit 402integrates the second exposure image 142 in a horizontal direction andoutputs the integrated value to the dividing unit 403.

The integrated value calculating unit 401 and the integrated valuecalculating unit 402, which respectively calculate the integrated valuesof the first exposure image 141 and the second exposure image, mayperform processing while targeting at all the pixels or may performprocessing while targeting at pixels within a predetermined area.

Further, it is also possible to employ a configuration where pixels atpositions where pixels of either the first exposure image 141 or thesecond exposure image 142 are saturated, are not used for integrationoperation. For example, a portion which does not include halation, ablack defect, a subject portion, or the like, can be set as a portion tobe used when integration is performed.

To the dividing unit 403, the integrated value (referred to as a firstintegrated value) calculated from the first exposure image 141 issupplied from the integrated value calculating unit 401, and theintegrated value (referred to as a second integrated value) calculatedfrom the second exposure image 142 is supplied from the integrated valuecalculating unit 402. The dividing unit 403 obtains a flicker ratio bydividing the supplied first integrated value by the second integratedvalue. The obtained flicker ratio 411 is supplied to the estimationoperating unit 404.

The estimation operating unit 404 respectively calculates the firstflicker component 381 and the second flicker component 382 from theflicker ratio 411. Here, calculation of the flicker ratio 411 will bedescribed.

It should be noted that the flicker ratio is a ratio of flickercomponents of respective images, and, description will be continued hereassuming that the flicker ratio is a ratio of the first flickercomponent 381 and the second flicker component 382. Further, when it isintended to remove noise having a predetermined cycle like flicker, aratio of noise components for each image corresponds to a flicker ratiodescribed below, and the ratio of noise components can be calculated ina similar manner to the flicker ratio described below.

<Calculation of Flicker Ratio>

Here, the flicker ratio will be described with reference to FIG. 19.Part A in FIG. 19 illustrates the first exposure image 141 and thesecond exposure image 142 which are examples of images in which thereare stripe patterns on the images due to influence of flicker.

One example of a result in the case where integration is performed in ahorizontal direction on an effective portion of the first exposure image141 is illustrated in an upper part of part B in FIG. 19. Further, oneexample of a result in the case where integration is performed in ahorizontal direction on an effective portion of the second exposureimage 142 is illustrated in a lower portion of part B in FIG. 19. Theeffective portions used for integration are portions which do notinclude halation, a black defect, a subject portion, or the like.

By obtaining a ratio for each of the same positions in the horizontaldirection for which integration is performed, it is possible to obtain aflicker ratio as illustrated in part C in FIG. 19.

By multiplying one integrated value by a gain so that exposure levelsbecome the same, subject components other than flicker components becomethe same between two images, and, thus, are cancelled. By this means,because it is only necessary to purely take into account only flickercomponents in the two images, it is possible to improve accuracy ofestimation of flicker components thereafter.

It should be noted that while, in FIG. 19, a flicker ratio is calculatedfor each ordinate of an image as one example, for example, when thenumber of cycles of the flicker is known, it is also possible to employa configuration where coordinates corresponding to the number of cyclesof the flicker are provided, and integrated values of the same phase areintegrated on the same coordinate.

Such calculation of the flicker ratio will be further described. Anobservation image affected by flicker can be expressed as follows whenexposure completion time on the top row of the exposure image is t, andthe exposure period is E.

I(x,y,t,E)=I ₀(x,y)×E×g _(t)(t+y,E)  (7)

In formula (7), I₀(x, y) is a value of a true image which is notaffected by flicker in a unit exposure period.

As described above, when the short-period exposure is the first exposureconditions, and the long-period exposure is the second exposureconditions, the exposure period and the exposure completion time aredefined as follows.

First exposure conditions: the exposure period is E₁, and the exposurecompletion time on the top row is t₁

Second exposure conditions: the exposure period is E₂, and the exposurecompletion time on the top row is t₂

Further, E₁≦E₂.

Further, an image picked up under the first exposure conditions and animage picked up under the second exposure conditions can be respectivelyexpressed with the following formula (8), when simply expressed usingthe above-described formula (7).

I ₁(x,y)=I(x,y,t ₁ ,E ₁)

I ₂(x,y)=I(x,y,t ₂ ,E ₂)  (8)

When definition is provided as described above, the flicker ratio can beobtained as follows.

The flicker ratio is defined as a ratio of the flicker component underthe first exposure conditions and the flicker component under the secondexposure conditions. The flicker ratio can be obtained from two exposureimages using the following formula (9).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 6} \right\rbrack & \; \\{{r_{12}(y)} = {\frac{E_{2} \times {\int_{x,{y \in \Omega}}^{\;}{{I_{1}\left( {x,y} \right)}{x}}}}{E_{1} \times {\int_{x,{y \in \Omega}}^{\;}{{I_{2}\left( {x,y} \right)}{x}}}} = \frac{g_{t}\left( {{y + t_{1}},E_{1}} \right)}{g_{t}\left( {{y + t_{2}},E_{2}} \right)}}} & (9)\end{matrix}$

However, in order to avoid influence of saturated pixels, integration isexecuted while an area (Ω) where neither of the first exposure image 141under the first exposure conditions and the second exposure image 142under the second exposure conditions is saturated is used as an area forwhich integration is performed.

Operation performed at each unit of the integrated value calculatingunit 401, the integrated value calculating unit 402 and the dividingunit 403 of the flicker estimating unit 371 illustrated in FIG. 18 isperformed based on operation in formula (9). The flicker ratio 411 iscalculated in this manner.

The following relational expression of formula (10) can be determinedusing formula (9).

r ₁₂(y)×g _(t)(y+t ₂ ,E ₂)=g ₁(y+t ₁ ,E ₁)  (10)

If the flicker ratio can be obtained based on formula (9) using formula(10), it is possible to obtain one flicker component (for example, thesecond flicker component 381) from the other flicker component (forexample, the second flicker component 382).

<First Embodiment of Flicker Suppression>

As the first embodiment, a case will be described where a flickercomponent is obtained using solution in real space. FIG. 20 is a diagramillustrating an internal configuration diagram of the estimationoperating unit 404 of the flicker estimating unit 371 within the flickercorrecting unit 312 in the first embodiment.

The estimation operating unit 404 illustrated in FIG. 20 is configuredto include a matrix generating unit 431, a matrix operating unit 432 anda flicker component converting unit 433. The matrix generating unit 431generates a matrix which will be described below from the inputtedflicker ratio 411, and the matrix operating unit 432 performs operationon the matrix. By performing operation on the matrix, the first flickercomponent 381 is generated.

The first flicker component 381 from the matrix operating unit 432 isalso supplied to the flicker component converting unit 433. The flickercomponent converting unit 433 generates the second flicker component 382from the supplied first flicker component 381.

In this embodiment, one flicker component is generated from the otherflicker component. While, here, description will be continued assumingthat the second flicker component 382 included in the second exposureimage 142 obtained upon long-period exposure (second exposureconditions) is generated from the first flicker component 381 obtainedfrom the first exposure image 141 obtained upon short-period exposure(first exposure conditions), it is also possible to employ aconfiguration where the first flicker component 381 is generated fromthe second flicker component 382.

It is possible to apply operation disclosed in Patent Application No.2012-90897 filed before the present application by the applicant of thepresent application, or the like, as operation for obtaining one flickercomponent from the other flicker component as described above. PatentApplication No. 2012-90897 describes that a flicker componentg_(t)(y+t₂, E₂) upon long-period exposure (second exposure conditions)can be expressed using a linear sum of a flicker component g_(t)(y+t₂,E₂) upon short-period exposure (first exposure conditions).

That is, the flicker component can be expressed as the following formula(11) when expressed using a vector.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 7} \right\rbrack & \; \\{\begin{bmatrix}{g_{t}\left( {{t_{2} + 0},E_{2}} \right)} \\\vdots \\{g_{t}\left( {{t_{2} + N - 1},E_{2}} \right)}\end{bmatrix} = {t\begin{bmatrix}{g_{t}\left( {{t_{1} + 0},E_{1}} \right)} \\\vdots \\{g_{t}\left( {{t_{1} + N - 1},E_{1}} \right)}\end{bmatrix}}} & (11)\end{matrix}$

In formula (11), a matrix t is operation in Patent Application No.2012-90897, which is expressed as a matrix. For simplification, formula(11) is expressed as follows. In formula (12), g₁ indicates the firstflicker component 381, and g₂ indicates the second flicker component382.

g ₂ =t _(g1)  (12)

The above-described formula (10) can be expressed as follows usingformula (11) and formula (12).

$\begin{matrix}{\mspace{79mu} \left\lbrack {{Math}.\mspace{14mu} 8} \right\rbrack} & \; \\{{{r_{12}(y)} \times {g_{t}\left( {{y + t_{2}},E_{2}} \right)}} = {\left. {g_{t}\left( {{y + t_{1}},E_{1}} \right)}\leftrightarrow{\begin{bmatrix}{r_{12}(0)} & \; & \; & 0 \\\; & {r_{12}(1)} & \; & \; \\\; & \; & \ddots & \; \\0 & \; & \; & {r_{12}\left( {N - 1} \right)}\end{bmatrix}g_{2}} \right. = {\left. g_{1}\leftrightarrow{rg}_{2} \right. = g_{1}}}} & (13)\end{matrix}$

In formula (13), a matrix r can be measured from an image and a matrix tcan be obtained from mutual relationship between the first exposureconditions and the second exposure conditions. Therefore, the firstflicker component 381 and the second flicker component 382 are obtainedfrom the following two formulas. A formula described in an upper part ofthe following formula (14) is formula (12), and a formula described in alower part is formula (13).

g ₂ =tg ₁

rg ₂ =g ₁  (14)

As solution 1, flicker components g₁ and g₂ are obtained by utilizingthat the following formula (15) can be derived from formula (14).

rtg ₁ =g ₁  (15)

If an eigenvector of an eigenvalue 1 of a matrix rt is obtained fromformula (15), it is possible to obtain the flicker component g₁ (firstflicker component 381) under the first exposure conditions (short-periodexposure). Typically, while the eigenvector has arbitrary property of aconstant factor, because an average value of the flicker components isapproximately 1, it is possible to uniquely obtain the first flickercomponent 381. If the first flicker component 381 can be obtained, it ispossible to obtain the second flicker component 382 from the firstflicker component 381 (for example, a technique disclosed in PatentApplication No. 2012-98097 can be applied).

As solution 2, formula (14) is expressed as one matrix as in thefollowing formula (16).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 9} \right\rbrack & \; \\{{\begin{bmatrix}t & {- I} \\I & {- r}\end{bmatrix}\begin{bmatrix}g_{1} \\g_{2}\end{bmatrix}} = 0} & (16)\end{matrix}$

It is only necessary to obtain the first flicker component 381 and thesecond flicker component 382 which satisfy formula (16) usingleast-squares estimation. In formula (16), t is a coefficient used forconverting a flicker component upon short-period exposure into a flickercomponent upon long-period exposure, and r is a flicker ratio expressedwith formula (9), and can be obtained from an image.

I, which is a pixel value, can be obtained from an image. Therefore, anumerical value within the first term in formula (16) is a value whichcan be obtained from an image, or the like. It can be seen from thisthat the first flicker component 381 and the second flicker component382 can be obtained from formula (16) using least-squares estimation.

It is possible to respectively obtain the first flicker component 381and the second flicker component 382 by applying either solution 1 orsolution 2.

If solution 1 is applied, the matrix generating unit 431 of theestimation operating unit 404 illustrated in FIG. 20 generates thematrix rt expressed with formula (15), and the matrix operating unit 432performs matrix operation using the generated matrix rt and generatesthe flicker component g₁ (first flicker component 381).

Specifically, the matrix operating unit 432 obtains the eigenvector ofthe eigenvalue 1 of the matrix rt. The obtained eigenvector becomes thefirst flicker component 381. It should be noted that an average value ofthe eigenvector is limited to be 1. The flicker component convertingunit 433 generates the second flicker component 382 from the firstflicker component 381 based on relational expression of g₂=tg₁ expressedwith formula (14).

If solution 2 is applied, the matrix generating unit 431 generates thematrix expressed with formula (16), and the matrix operating unit 432performs operation on the matrix. The first flicker component 381 isgenerated through the operation by the matrix operating unit 432, andthe second flicker component 382 is generated by the flicker componentconverting unit 433. For example, the matrix operating unit 432 can beconfigured to obtain a predetermined function using least-squares methodto calculate the first flicker component 381, and the flicker componentconverting unit 433 can be configured to calculate the second flickercomponent 382 from the predetermined function and the first flickercomponent 381.

It should be noted that while FIG. 20 illustrates the matrix operatingunit 432 and the flicker component converting unit 433 as differentblocks, the matrix operating unit 432 and the flicker componentconverting unit 433 may be the same block. In other words, in order toclearly illustrate that after one flicker component is calculated, theother flicker component is calculated, the matrix operating unit 432 andthe flicker component converting unit 433 are separately illustrated anddescribed.

However, it is also possible to employ a configuration where the matrixoperating unit 432 and the flicker component converting unit 433 areconfigured as one block, one flicker component is generated and theother flicker component is generated from the flicker component withinthe block, and the first flicker component 381 and the second flickercomponent 382 are finally outputted from the block.

In this manner, it is possible to apply solution in real space tocalculate respective flicker components of two images under differentexposure conditions. Therefore, it is possible to correct the respectiveflicker components of the two images under different exposure conditionsand generate one high dynamic range image from the two image for whichthe flicker components are corrected.

<Second Embodiment of Flicker Suppression>

Next, as the second embodiment, a case will be described where flickercomponents are obtained using solution in complex space. FIG. 21 is adiagram illustrating an internal configuration example of an estimationoperating unit 404′ of the flicker estimating unit 371 within theflicker correcting unit 312 in the second embodiment. It should be notedthat, here, in order to differentiate the estimation operating unit 404′from the estimation operating unit 404 in the first embodimentillustrated in FIG. 20, the estimation operating unit 404′ in the secondembodiment will be described with a dash mark being added.

The estimation operating unit 404′ illustrated in FIG. 21 is configuredto include a Fourier series transforming unit 461, a matrix generatingunit 462, a matrix operating unit 463, a Fourier series inversetransforming unit 464, a flicker component converting unit 465 and aFourier series inverse transforming unit 466.

The flicker ratio 411 is supplied to the Fourier series transformingunit 461 of the estimation operating unit 404′. The Fourier seriestransforming unit 461 performs Fourier series expansion on the suppliedflicker ratio 411. A reference frequency of the Fourier series of theFourier series transforming unit 461 can be made the same as a frequencyof the flicker.

For example, when flicker is caused by a fluorescent light, or the like,the reference frequency of the Fourier series of the Fourier seriestransforming unit 461 is set at 100 Hz or 120 Hz. Further, when it isintended to remove cyclic noise, a frequency appropriate for a cycle ofthe noise is set as the reference frequency of the Fourier series.

The matrix generating unit 462 obtains a matrix RT which will bedescribed later. The matrix operating unit 463 obtains an eigenvector ofan eigenvalue 1 of the matrix RT generated at the matrix generating unit462 under the conditions that G₁(0)=1 and

[Math. 10]

G ₁(ω) and G ₁(ω)

become a complex conjugate.

The value calculated at the matrix operating unit 463 is a valueobtained by performing Fourier series expansion on the first flickercomponent 381. By performing Fourier series inverse transform on thisvalue at the Fourier series inverse transforming unit 464, the firstflicker component 381 can be obtained.

A value obtained by performing Fourier series expansion on the firstflicker component 381 is also supplied to the flicker componentconverting unit 465 from the matrix operating unit 463. The flickercomponent converting unit 465 generates a value obtained by convertingthe second flicker component 382 in Fourier series by multiplying aconversion coefficient T₁₂(ω) by a value obtained by performing Fourierseries expansion on the first flicker component.

The value converted at the flicker component converting unit 465 issupplied to the Fourier series inverse transforming unit 466, where thevalue is subjected to Fourier series inverse transform to be made thesecond flicker component 382.

In this manner, by obtaining solution in frequency space (complexspace), it is possible to obtain a flicker component with lessoperation. This operation will be further described.

As described above, the following relational expression (10) holdsbetween the flicker ratio and the flicker components.

r ₁₂(y)×g _(t)(y+t ₂ ,E ₂)=g ₁(y+t ₁ ,E ₁)  (10)

When Fourier transform is performed on both sides of this formula (10),because multiplication becomes convolution operation, the followingformula (17) can be obtained. It should be noted that, here, basically,symbols in real space are expressed using small letters, while symbolsin frequency space are expressed using capital letters.

[Math. 11]

R ₁₂(ω){circle around (×)}G ₂(ω)=G ₁(ω)  (17)

In formula (17), G₁(ω) indicates a value obtained by performing Fouriertransform on the first flicker component 381, while G₂(ω) indicates avalue obtained by performing Fourier transform on the second flickercomponent 382. Further, as will be described later, G₁(ω) and G₂(ω) areexpressed using a value T₁₂(ω) obtained from the first exposureconditions and the second exposure conditions as in the followingformula (18).

[Math. 12]

G ₂(ω)=T ₁₂(ω)×G ₁(ω)  (18)

The following formula (19) can be obtained from formula (17) and formula(18).

[Math. 13]

R ₁₂(ω){circle around (×)}T ₁₂(ω)×G ₁(ω)=G ₁(ω)  (19)

Further, when formula (19) is expressed with a matrix, becauseconvolution operation becomes a cyclic matrix, formula (19) can beexpressed with the following formula (20).

$\begin{matrix}{\mspace{79mu} \left\lbrack {{Math}.\mspace{14mu} 14} \right\rbrack} & \; \\{{\begin{bmatrix}R_{0} & {\overset{\_}{R}}_{1} & \ldots & {\overset{\_}{R}}_{M} & \; & \; & 0 \\R_{1} & R_{0} & \ddots & \ddots & \ddots & \; & \; \\\vdots & \ddots & R_{0} & {\overset{\_}{R}}_{1} & \ddots & \ddots & \; \\R_{M} & \ddots & \ddots & R_{0} & \ddots & \ddots & {\overset{\_}{R}}_{M} \\\; & \ddots & \ddots & R_{1} & R_{0} & \ddots & \vdots \\\; & \; & \ddots & \ddots & \ddots & R_{0} & {\overset{\_}{R}}_{1} \\0 & \; & \; & R_{M} & \ldots & R_{1} & R_{0}\end{bmatrix}\begin{bmatrix}{\overset{\_}{T}}_{M} & \; & \; & \; & \; & \; & \; \\\; & \ddots & \; & \; & \; & 0 & \; \\\; & \; & {\overset{\_}{T}}_{1} & \; & \; & \; & \; \\\; & \; & \; & T_{0} & \; & \; & \; \\\; & \; & \; & \; & T_{1} & \; & \; \\\; & {0\;} & \; & \; & \; & \ddots & \; \\\; & \; & \; & \; & \; & \; & T_{M}\end{bmatrix}}{\quad{\left\lbrack \begin{matrix}\overset{\_}{G_{1}(M)} \\\vdots \\\overset{\_}{G_{1}(1)} \\{G_{1}(0)} \\{G_{1}(1)} \\\vdots \\{G_{1}(M)}\end{matrix} \right\rbrack = \begin{bmatrix}\overset{\_}{G_{1}(M)} \\\vdots \\\overset{\_}{G_{1}(1)} \\{G_{1}(0)} \\{G_{1}(1)} \\\vdots \\{G_{1}(M)}\end{bmatrix}}}} & (20)\end{matrix}$

In formula (12), R₁₂(ω) and T₁₂(ω) are respectively abbreviated as R_(ω)and T_(ω). Further, because a value of a negative frequency becomes acomplex conjugate of a value of a positive frequency, the value of thenegative frequency is expressed as follows.

[Math. 15]

G ₁(−ω)= G ₁(ω)

Formula (19) or formula (20) is expressed as in the following formula(21).

RTG ₁ =G ₁  (21)

It is only necessary to obtain the eigenvector of the eigenvalue 1 ofthe matrix RT under the conditions that

[Math. 16]

G ₁(ω) and G ₁(ω)

become a complex conjugate.

However, typically, an eigenvector can be the same eigenvector even ifthe eigenvector is multiplied by an arbitrary constant. However, becausethe average value of the flicker components is 1, and

G ₁(0)=1,

and is known, it is possible to uniquely obtain solution.

Further, a method for estimating the first flicker component 381 and thesecond flicker component 382 in complex space will be further described.

The flicker component can be expressed as in the following formula (22).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 17} \right\rbrack & \; \\\begin{matrix}{{g\left( {t,E} \right)} = \frac{F_{A}\left( {t,E} \right)}{F_{D}\left( {t,E} \right)}} \\{= \frac{\int_{t - E}^{t}{{f(\tau)}\ {\tau}}}{\int_{t - E}^{t}{f_{D}\ {\tau}}}} \\{= {\frac{1}{E}{\int_{t - E}^{t}{\frac{f(\tau)}{f_{d}}\ {\tau}}}}} \\{= {\frac{1}{E}{\int_{t - E}^{t}{{f^{\prime}(\tau)}\ {\tau}}}}}\end{matrix} & (22)\end{matrix}$

Formula (22) can be expressed as in the following formula (23) usingconvolution operation of fluctuation of a light source and a shutterfunction.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 18} \right\rbrack & \; \\{{{g\left( {t,E} \right)} = {{f^{\prime}(t)} \otimes {s\left( {t,E} \right)}}}\left\{ \begin{matrix}{{f^{\prime}(t)} = \frac{f(t)}{f_{D}}} \\{{s\left( {t,E} \right)} = {\frac{1}{E}{{rect}\left( {\frac{1}{E} - \frac{1}{2}} \right)}}}\end{matrix} \right.} & (23)\end{matrix}$

In formula (23), f′(t) is a value obtained by normalizing fluctuation ofthe light source f(t) so that an average value becomes 1 as illustratedin part A in FIG. 22. Further, in formula (23), a shutter function s(t,E) is a function as indicated in part B in FIG. 22. The functionindicated in part B in FIG. 22 is a function having a value of 1/E whiletime is from 0 to E and having a value of 0 in other time.

Operation is performed after the flicker component is converted infrequency space. When gt(t, E) is expressed in frequency space, becauseconvolution operation becomes multiplication, the following formula (24)can be obtained.

[Math. 19]

G(ω,E)=F′(ω)×S(ω,E)  (24)

Further, when the flicker component under the first exposure conditionsand the flicker component under the second exposure conditions expressedin frequency space are respectively defined as G₁(ω) and G₂(ω), G₁(ω)and G₂(ω) can be respectively expressed as in the following formula(25).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 20} \right\rbrack & \; \\\left\{ \begin{matrix}{{G_{1}(\omega)} = {{F^{\prime}(\omega)} \times {S\left( {\omega,E_{1}} \right)} \times {\exp \left( {2{\pi \omega}\; t_{1}} \right)}}} \\{{G_{2}(\omega)} = {{F^{\prime}(\omega)} \times {S\left( {\omega,E_{2}} \right)} \times {\exp \left( {2{\pi \omega}\; t_{2}} \right)}}}\end{matrix} \right. & (25)\end{matrix}$

G₂(ω) can be expressed as in the following formula (26) from formula(25) using G₁(ω).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 21} \right\rbrack & \; \\{{{G_{2}(\omega)} = {{T_{12}(\omega)} \times {G_{1}(\omega)}}}{{{where}\mspace{14mu} {T_{12}(\omega)}} = {\frac{S\left( {\omega,E_{2}} \right)}{S\left( {\omega,E_{1}} \right)}\exp \left\{ {{\omega}\left( {t_{2} - t_{1}} \right)} \right\}}}} & (26)\end{matrix}$

In formula (26), s(ω, E) is a shutter function s(t, E) expressed infrequency space, and can be expressed as in the following formula (27).

$\begin{matrix}{\mspace{79mu} \left\lbrack {{Math}.\mspace{14mu} 22} \right\rbrack} & \; \\{\mspace{79mu} {{{s\left( {t,E} \right)} = {\frac{1}{E}{{rect}\left( {\frac{t}{E} - \frac{1}{2}} \right)}}}{{S\left( {\omega,E} \right)} = {{{\frac{1}{E} \cdot {{sinc}\left( {E\; \omega} \right)}}{\exp \left( {{- 2}\pi \; {\omega} \times \frac{1}{2}E} \right)}} = {{{sinc}\left( {E\; \omega} \right)}{\exp \left( {{- {\pi \omega}}\; E} \right)}}}}}} & (27)\end{matrix}$

As can be seen from formula (26), a value (G₂(ω)) obtained by performingFourier series expansion on the second flicker component 382 can becalculated from a value (G₁(ω)) obtained by performing Fourier seriesexpansion on the first flicker component 381. By operation based onformula (26) being performed at the flicker component converting unit465 (FIG. 21), a value obtained by performing Fourier series expansionon the second flicker component 382 is calculated.

Further, the matrix generating unit 462 generates the matrix RT informula (20) and formula (21), and the matrix operating unit 462calculates the eigenvector of the eigenvalue 1 of the matrix RT. Becausethis calculated eigenvector is a value obtained by performing Fourierseries expansion on the first flicker component 381, by the Fourierseries inverse transform being performed at the Fourier series inversetransforming unit 464, the first flicker component 381 is calculated.

On the other hand, as described above, the flicker component convertingunit 465 performs operation based on formula (26) to thereby convert thefirst flicker component 381 into the second flicker component 382.Because this converted value is a value obtained by performing Fourierseries expansion on the second flicker component 382, by Fourier seriesinverse transform being performed at the Fourier series inversetransforming unit 466, the second flicker component 382 is calculated.

By applying solution in complex space in this manner, it is possible tocalculate respective flicker components of two images under differentexposure conditions. Therefore, it is possible to correct the respectiveflicker components of the two images under different exposure conditionsand generate one high dynamic range image from the two images for whichthe flicker components are corrected.

<Third Embodiment of Flicker Suppression>

In the first embodiment and the second embodiment, description has beenprovided using a case as an example where the flicker components aresuppressed by calculating the first flicker component 381 and the secondflicker component 382 using the flicker ratio between the flickercomponent of the image picked up while being exposed under the firstexposure conditions and the flicker component of the image picked upwhile being exposed under the second exposure conditions.

Next, as the third embodiment, a case will be described as an examplewhere the first flicker component 381 and the second flicker component382 are calculated without using the flicker ratio.

FIG. 23 is a diagram illustrating a configuration of a flickerestimating unit 371′. In order to differentiate the flicker estimatingunit 371′ from the flicker estimating unit 371 illustrated in FIG. 18,the flicker estimating unit 371′ illustrated in FIG. 23 will bedescribed with a dash mark being added. Further, the flicker estimatingunit 371′ illustrated in FIG. 23 is the flicker estimating unit 371which configures the flicker correcting unit 312 illustrated in FIG. 17.

The flicker estimating unit 371′ illustrated in FIG. 23 is configuredwith an integrated value calculating unit 501, an integrated valuecalculating unit 502, a Fourier series transforming unit 503, a Fourierseries transforming unit 504, an estimation operating unit 505, aFourier series inverse transforming unit 506 and a Fourier seriesinverse transforming unit 507.

The first exposure image 141 is supplied to the integrated valuecalculating unit 501, and the second exposure image 142 is supplied tothe integrated value calculating unit 502. The integrated valuecalculating unit 501 integrates the first exposure image 141 in ahorizontal direction and outputs the integrated value to the Fourierseries transforming unit 503. In a similar manner, the integrated valuecalculating unit 502 integrates the second exposure image 142 in ahorizontal direction and outputs the integrated value to the Fourierseries transforming unit 504.

While the integrated value calculating unit 501 and the integrated valuecalculating unit 502 respectively calculate the integrated values of thefirst exposure image 141 and the second exposure image, the integratedvalue calculating unit 501 and the integrated value calculating unit 502may perform processing while targeting at all the pixels or may performprocessing while targeting at pixels within a predetermined area.

Further, integration operation may not be used for pixels at positionswhere pixels of either the first exposure image 141 or the secondexposure image 142 are saturated. For example, a portion which does notinclude halation, a black defect, a subject portion, or the like, can beset as a portion to be used for integration.

The Fourier series transforming unit 503 performs Fourier seriesexpansion by multiplying the integrated value of the first exposureimage 141 by an appropriate window function and obtains frequencyinscription J₁(ω) (which will be described later in details) of thefirst exposure image 141. In a similar manner, the Fourier seriestransforming unit 504 performs Fourier series expansion by multiplyingthe integrated value of the second exposure image 142 by an appropriatewindow function and obtains frequency inscription J₂(ω) of the secondexposure image 142. It should be noted that the window function istypically used in Fourier transform, or the like, and, for example, ahann window can be used.

To the estimation operating unit 505, the frequency inscription J₁(ω) ofthe first exposure image 141 is supplied from the Fourier seriestransforming unit 503, and the frequency inscription J₂(ω) of the secondexposure image 142 is supplied from the Fourier series transforming unit504. The estimation operating unit 505 generates a matrix Q and obtainsa fluctuation component F′ of the light source. Further, the estimationoperating unit 505 obtains a value obtained by performing Fourier seriesexpansion on the first flicker component 381 and a value obtained byperforming Fourier series expansion on the second flicker component 382.

The value obtained by performing Fourier series expansion on the firstflicker component 381 obtained at the estimation operating unit 505 issupplied to the Fourier series inverse transforming unit 506, where thevalue is subjected to Fourier series inverse transform to be made thefirst flicker component 381. In a similar manner, the value obtained byperforming Fourier series expansion on the second flicker component 382obtained at the estimation operating unit 505 is supplied to the Fourierseries inverse transforming unit 507, where the value is subjected toFourier series inverse transform to be made the second flicker component382.

Also in the third embodiment, as in the second embodiment, by obtainingsolution in frequency space (complex space), it is possible to obtainflicker components with less operation. This operation will be furtherdescribed.

FIG. 24 illustrates processing of the integrated value calculating units501 and 502, and the Fourier series transforming units 503 and 504, andillustrates process of obtaining J₁(ω) and J₂(ω) in frequency space ofimages under respective exposure conditions. Part A in FIG. 24illustrates the first exposure image 141 and the second exposure image142 and illustrates an example where the images include stripe patternsdue to influence of flicker.

One example of a result in the case where an effective portion of thefirst exposure image 141 is integrated in a horizontal direction will beillustrated in an upper part of part B in FIG. 24. Further, one exampleof a result in the case where an effective portion of the secondexposure image 142 is integrated in a horizontal direction will beillustrated in a lower part of part B in FIG. 24. Effective portionsused for integration are portions which do not include halation, a blackdefect, a subject portion, or the like.

The calculated integrated value is multiplied by an appropriate windowfunction, and subjected to Fourier series expansion. As Fourier series,values of frequencies from 0×ωk to 2M×ωk are calculated. As a result,the frequency inscription J₁(ω) of the first exposure image 141 and thefrequency inscription J₂(ω) of the second exposure image 142 arerespectively obtained (part C in FIG. 24).

Such operation will be further described. The first exposure image 141picked up under the first exposure conditions and the second exposureimage 142 picked up under the second exposure conditions arerespectively expressed as follows.

I ₁(x,y)=I ₀(x,y)×E ₁ ×g _(t)(t ₁ +y,E ₁)

I ₂(x,y)=I ₀(x,y)×E ₂ ×g _(t)(t ₂ +y,E ₂)  (28)

From formula (28), the following formula (29) holds.

I ₁(x,y)×E ₂ ×g _(t)(t ₂ −y,E ₂)−I ₂(x,y)×E ₁ ×g _(t)(t ₁ +y,E₁)=0  (29)

Formula (29) is expressed as the following formula (30) when expressedin frequency space.

[Math. 23]

E ₂ J ₁(ω){circle around (×)}G ₂(ω)−E ₁ J ₂(ω){circle around (×)}G₁(ω)=0  (30)

In formula (30), J₁(ω) is a value obtained by integrating I₁(x, y) in ahorizontal direction, that is, a value expressing a value obtained byintegrating the first exposure image 141 in a horizontal direction infrequency space. In a similar manner, J₂(ω) is a value obtained byintegrating I₂(x, y) in a horizontal direction, that is, a valueexpressing a value obtained by integrating the second exposure image 142in a horizontal direction in frequency space.

When the flicker component G₁(ω) and the flicker component G₂(ω) areassigned in formula (30), the following formula (31) is obtained. [Math.24]

E ₂ J ₁(ω){circle around (×)}{S(ω,E ₂)×exp(2πωt ₂)×F′(ω)}−E ₁ J₂(ω){circle around (×)}{S(ω,E ₁)×exp(2πωt ₁)×F′(ω)}=0   (31)

Formula (31) is a linear formula and can be expressed as in thefollowing formula (32) as a matrix.

E ₂ J ₁ S ₂ P ₂ F′−E ₁ J ₂ S ₁ P ₁ F′−0

(E ₂ J ₁ S ₂ P ₂ −E ₁ J ₂ S ₁ P ₁)F′=0  (32)

J₁, S₁, P₁ and F′ in formula (32) are matrices respectively expressed inthe following formula (33) to formula (36). It should be noted thatbecause J₂, S₂ and P₂ can be expressed in a similar manner, expressionwill be omitted here.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 25} \right\rbrack & \; \\{J_{1} = \begin{bmatrix}{J_{1}\left( {0 \times \omega_{f}} \right)} & \ddots & \ddots & \overset{\_}{J_{1}\left( {M \times \omega_{f}} \right)} & \ddots & \ddots & \overset{\_}{J_{1}\left( {2M \times \omega_{f}} \right)} \\{J_{1}\left( {1 \times \omega_{f}} \right)} & \ddots & \ddots & \ddots & \ddots & \ddots & \ddots \\\ddots & \ddots & \ddots & \overset{\_}{J_{1}\left( {1 \times \omega_{f}} \right)} & \ddots & \ddots & \ddots \\\ddots & \ddots & \ddots & {J_{1}\left( {0 \times \omega_{f}} \right)} & \ddots & \ddots & \ddots \\\ddots & \ddots & \ddots & {J_{1}\left( {1 \times \omega_{f}} \right)} & \ddots & \ddots & \ddots \\\ddots & \ddots & \ddots & \ddots & \ddots & \ddots & \overset{\_}{J_{1}\left( {1 \times \omega_{f}} \right)} \\{J_{1}\left( {2M \times \omega_{f}} \right)} & \ddots & \ddots & {J_{1}\left( {M \times \omega_{f}} \right)} & \ddots & \ddots & {J_{1}\left( {0 \times \omega_{f}} \right)}\end{bmatrix}} & (33) \\\left\lbrack {{Math}.\mspace{14mu} 26} \right\rbrack & \; \\{S_{1} = \begin{bmatrix}\overset{\_}{S\left( {{M \times \omega_{k}},E_{1}} \right)} & \; & \; & \; & \; & \; & \; \\\; & \ddots & \; & \; & \; & 0 & \; \\\; & \; & \overset{\_}{S\left( {{1 \times \omega_{k}},E_{1}} \right)} & \; & \; & \; & \; \\\; & \; & \; & {S\left( {{0 \times \omega_{k}},E_{1}} \right)} & \; & \; & \; \\\; & \; & \; & \; & {S\left( {{1 \times \omega_{k}},E_{1}} \right)} & \; & \; \\\; & 0 & \; & \; & \; & \ddots & \; \\\; & \; & \; & \; & \; & \; & {S\left( {{M \times \omega_{k}},E_{1}} \right)}\end{bmatrix}} & (34) \\\left\lbrack {{Math}.\mspace{14mu} 27} \right\rbrack & \; \\{P_{1} = \begin{bmatrix}{\exp \left( {{- M} \times 2{\pi\omega}_{f}t_{1}} \right)} & \; & \; & \; & \; & \; & \; \\\; & \ddots & \; & \; & \; & 0 & \; \\\; & \; & {\exp \left( {{- 1} \times 2{\pi\omega}_{f}t_{1}} \right)} & \; & \; & \; & \; \\\; & \; & \; & {\exp \left( {0 \times 2{\pi\omega}_{f}t_{1}} \right)} & \; & \; & \; \\\; & \; & \; & \; & {\exp \left( {1 \times 2{\pi\omega}_{f}t_{1}} \right)} & \; & \; \\\; & 0 & \; & \; & \; & \ddots & \; \\\; & \; & \; & \; & \; & \; & {\exp \left( {M \times 2{\pi\omega}_{f}t_{1}} \right)}\end{bmatrix}} & (35) \\\left\lbrack {{Math}.\mspace{14mu} 28} \right\rbrack & \; \\{F = \begin{bmatrix}\overset{\_}{F^{\prime}\left( {M \times \omega_{f}} \right)} \\\vdots \\\overset{\_}{F^{\prime}\left( {1 \times \omega_{f}} \right)} \\{F^{\prime}\left( {0 \times \omega_{f}} \right)} \\{F^{\prime}\left( {1 \times \omega_{f}} \right)} \\\vdots \\{F^{\prime}\left( {M \times \omega_{f}} \right)}\end{bmatrix}} & (36)\end{matrix}$

In formula (33) to formula (36), ω_(f) is a fundamental frequency offlicker, and, normally, is a value corresponding to 100 Hz or 120 Hz.When noise in a predetermined cycle is removed, ω_(f) is made afrequency corresponding to the cycle. Further, it is assumed that awaveform of the light source includes a frequency of up to M times ofthe fundamental frequency ω_(f). In the case of a typical light source,M is 1. Further, there is a case where M is greater than 1 according tolight sources because some light sources have high-frequency timevarying components.

Here, formula (32) is expressed as in the following formula (33).

QF′=0

Q=E ₂ J ₁ S ₂ P ₂ −E ₁ J ₂ S ₁ P ₁  (37)

In Q in formula (37), E indicates an exposure period, and J indicates anexposure image expressed in frequency space after being integrated in ahorizontal direction. Further, S indicates a shutter function expressedon frequency space and the shutter function expressed on frequency spacedescribed with reference to FIG. 22. Further, P is e×p(2πωt). Becausethese values are obtained from the exposure image and the exposureperiod, Q is known.

Because Q is known, F′ can be obtained from formula (37). When F′ isobtained, a flicker component can be obtained from the following formula(38).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 29} \right\rbrack & \; \\\left\{ \begin{matrix}{{G_{1}(\omega)} = {{F^{\prime}(\omega)} \times {S\left( {\omega,E_{1}} \right)} \times {\exp \left( {2{\pi \omega}\; t_{1}} \right)}}} \\{{G_{2}(\omega)} = {{F^{\prime}(\omega)} \times {S\left( {\omega,E_{2}} \right)} \times {\exp \left( {2{\pi \omega}\; t_{2}} \right)}}}\end{matrix} \right. & (38)\end{matrix}$

G₁(ω) in formula (38) is the first flicker component 381 of the firstexposure image 141 picked up under the first exposure conditions,expressed in frequency space, and G₂(ω) is the second flicker component382 of the second exposure image 142 picked up under the second exposureconditions, expressed in frequency space.

By respectively converting the flicker component 381 and the secondflicker component 382 in frequency space obtained using formula (38)into flicker components in real space, the first flicker component 381and the second flicker component 382 can be respectively obtained.

The estimation operating unit 505 (FIG. 23) of the flicker estimatingunit 371′ generates a matrix Q in formula (37) and obtains a fluctuationcomponent F′ of the light source. Then, the estimation operating unit505 performs operation based on formula (38) to generate a flickercomponent in frequency space.

In this manner, it is possible to calculate respective flickercomponents of two images under different exposure conditions by applyingsolution in complex space without using a flicker ratio. Therefore, itis possible to correct the respective flicker components of the twoimages under different exposure conditions and generate one high dynamicrange image from the two images for which the flicker components arecorrected.

While, in the above-described embodiments, when one image is picked up,as described above, a case has been described as an example where animage is picked up using short-period exposure (first exposureconditions) and long-period exposure (second exposure conditions) at thesame time, the present technology can be also applied to a case where animage upon short-period exposure and an image upon long-period exposureare acquired by alternately picking up a short-period exposure image anda long-period exposure image with normal pixels without separatingpixels for short-period exposure from pixels for long-period exposure.

In this case, because imaging timings are different, the presenttechnology can be applied by using a matrix which takes into account theimaging timings as a matrix to be used for operation upon theabove-described flicker correction.

Further, while, in the above-described examples, the imaging apparatuswhich picks up images with two types of exposure periods of short-periodexposure and long-period exposure has been described, the presenttechnology can be also applied to an imaging apparatus in which pickedup images with three or more types of exposure periods are combined.

When picked up images with three or more types of exposure periods arecombined, for example, it is possible to estimate the first flickercomponent from the first exposure image and the second exposure image,and convert the first flicker component into the third flicker componentto estimate the third flicker component. Further, it is also possible toobtain solution by generating a matrix in which all the first exposureimage, the second exposure image and the third exposure image arecombined.

Further, while, in the above-described embodiments, a case has beendescribed as an example where the flicker components are obtained usingexposure images respectively picked up with two different types ofexposure periods, the present technology can be also applied to a casewhere the flicker components are obtained using an exposure image pickedup with one type of an exposure period.

To obtain the flicker components using the exposure image picked up withone type of an exposure period, an image in the first frame is used asan exposure image picked up under the first exposure conditions in theabove-described embodiments, and an image in the second frame is used asan exposure image picked up under the second exposure conditions in theabove-described embodiments.

It should be noted that, in this case, because there is a possibilitythat the flicker components cannot be obtained under conditions where aspeed interval of imaging is the integral multiple of a cycle of theflicker, for example, the flicker components may be obtained using threeframes by picking up three frames, and making an imaging intervalbetween the first frame and the second frame different from an imaginginterval between the second frame and the third frame.

<Recording Medium>

The above-described series of processing can be executed with hardwareor can be executed with software. If the series of processing areexecuted with software, a program constituting software is installed ina computer. Here, the computer includes a computer incorporated intodedicated hardware, and, for example, a general-purpose personalcomputer which can execute various kinds of functions by various kindsof programs being installed.

FIG. 25 is a block diagram illustrating a configuration example ofhardware of a computer which executes the above-described series ofprocessing using a program. In the computer, a central processing unit(CPU) 1101, a read only memory (ROM) 1102 and a random access memory(RAM) 1103 are connected to one another via a bus 1104. An input/outputinterface 1105 is further connected to the bus 1104. An input unit 1106,an output unit 1107, a storage unit 1108, a communication unit 1109 anda drive 1110 are connected to the input/output interface 1105.

The input unit 1106 is formed with a keyboard, a mouse, a microphone, orthe like. The output unit 1107 is formed with a display, a speaker, orthe like. The storage unit 1108 is formed with a hard disc, anon-volatile memory, or the like. The communication unit 1109 is formedwith a network interface, or the like. The drive 1110 drives removablemedium 1111 such as a magnetic disc, an optical disc, a magnetoopticaldisc and a semiconductor memory.

In the computer configured as described above, the above-describedseries of processing are executed by, for example, the CPU 1101 loadinga program stored in the storage unit 1108 to the RAM 1103 via theinput/output interface 1105 and the bus 1104 and executing the program.

The program executed by the computer (CPU 1101) can be provided by, forexample, being recorded in the removable medium 1111 as package medium.Further, the program can be provided via wired or wireless transmissionmedium such as a local area network, the Internet and digital satellitebroadcasting.

In the computer, the program can be installed in the storage unit 1108via the input/output interface 1105 by the removable medium 1111 beingloaded to the drive 1110. Further, the program can be received at thecommunication unit 1109 via the wired or wireless transmission mediumand installed in the storage unit 1108. In addition, the program can beinstalled in advance in the ROM 1102 or the storage unit 1108.

It should be noted that the program executed by the computer may be aprogram which causes processing to be performed in time series in theorder described in the present specification or may be a program whichcauses processing to be performed in parallel or at a necessary timingsuch as upon calling.

Further, in the present specification, a system indicates the wholeapparatus configured with a plurality of apparatuses.

It should be noted that the advantageous effects described in thepresent specification are merely examples, and may be other advantageouseffects.

It should be noted that the embodiments of the present technology arenot limited to the above-described embodiments, and can be modified invarious manners without departing from the gist of the presenttechnology.

Additionally, the present technology may also be configured as below.

(1)

An image processing apparatus including:

an estimating unit configured to estimate cyclic noise componentsincluded in each image picked up under different exposure conditions foreach image,

wherein the estimating unit estimates the cyclic noise components foreach image through operation utilizing mutual relationship between thenoise components under the exposure conditions.

(2)

The image processing apparatus according to (1),

wherein the cyclic noise is flicker.

(3)

The image processing apparatus according to (1) or (2),

wherein the mutual relationship between the noise components isexpressed with a shutter function of the exposure conditions infrequency space.

(4)

The image processing apparatus according to any of (1) to (3),

wherein the estimating unit estimates the noise components using a valueobtained by integrating the images, multiplying a predetermined windowfunction and performing Fourier series expansion.

(5)

The image processing apparatus according to (4),

wherein the integration is performed in a horizontal direction for aportion not saturated in any of the images.

(6)

The image processing apparatus according to any of (1) to (5),

wherein the estimating unit obtains the noise components in frequencyspace by obtaining a matrix Q where QF=0 when a fluctuation component ofa light source is F and obtaining the fluctuation component F, and

wherein the estimating unit estimates the noise components for eachimage by performing Fourier series inverse transform on the noisecomponents in the frequency space.

(7)

The image processing apparatus according to (1),

wherein the mutual relationship between the noise components isexpressed with a ratio obtained by integrating the images and performingdivision for each row of the images.

(8)

The image processing apparatus according to (7),

wherein the integration is performed in a horizontal direction for aportion not saturated in any of the images.

(9)

The image processing apparatus according to (7),

wherein the estimating unit obtains an eigenvector of an eigenvalue 1 ofa matrix RT where R is a matrix obtained by performing Fourier seriesexpansion on the ratio and T is a matrix obtained from the exposureconditions, and sets the eigenvector as a value obtained by performingFourier series expansion on the noise components of the images.

(10)

The image processing apparatus according to (9),

wherein the noise components of the images are calculated by performingFourier series inverse transform on the eigenvector.

(11)

The image processing apparatus according to (10),

wherein a value obtained by performing Fourier series expansion on thenoise components of an image different from an image for which the noisecomponents have been calculated is calculated by multiplying theeigenvector by a coefficient obtained from the exposure conditions, and

wherein the noise components of the images are calculated by performingFourier series inverse transform on the value obtained by performingFourier series expansion.

(12)

The image processing apparatus according to (7),

wherein the estimating unit generates a matrix RT in the followingformula,

$\begin{matrix}{\mspace{79mu} \left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack} & \; \\{{\begin{bmatrix}R_{0} & {\overset{\_}{R}}_{1} & \ldots & {\overset{\_}{R}}_{M} & \; & \; & 0 \\R_{1} & R_{0} & \ddots & \ddots & \ddots & \; & \; \\\vdots & \ddots & R_{0} & {\overset{\_}{R}}_{1} & \ddots & \ddots & \; \\R_{M} & \ddots & \ddots & R_{0} & \ddots & \ddots & {\overset{\_}{R}}_{M} \\\; & \ddots & \ddots & R_{1} & R_{0} & \ddots & \vdots \\\; & \; & \ddots & \ddots & \ddots & R_{0} & {\overset{\_}{R}}_{1} \\0 & \; & \; & R_{M} & \ldots & R_{1} & R_{0}\end{bmatrix}\begin{bmatrix}{\overset{\_}{T}}_{M} & \; & \; & \; & \; & \; & \; \\\; & \ddots & \; & \; & \; & 0 & \; \\\; & \; & {\overset{\_}{T}}_{1} & \; & \; & \; & \; \\\; & \; & \; & T_{0} & \; & \; & \; \\\; & \; & \; & \; & T_{1} & \; & \; \\\; & {0\;} & \; & \; & \; & \ddots & \; \\\; & \; & \; & \; & \; & \; & T_{M}\end{bmatrix}}{\quad{\left\lbrack \begin{matrix}\overset{\_}{G_{1}(M)} \\\vdots \\\overset{\_}{G_{1}(1)} \\{G_{1}(0)} \\{G_{1}(1)} \\\vdots \\{G_{1}(M)}\end{matrix} \right\rbrack = \begin{bmatrix}\overset{\_}{G_{1}(M)} \\\vdots \\\overset{\_}{G_{1}(1)} \\{G_{1}(0)} \\{G_{1}(1)} \\\vdots \\{G_{1}(M)}\end{bmatrix}}}} & \;\end{matrix}$

where R is the ratio, T is the coefficient obtained from the exposureconditions and G is the noise components of the images, and obtains thenoise components of the images.(13)

The image processing apparatus according to (7),

wherein the estimating unit obtains an eigenvector of an eigenvalue 1 ofa matrix rt where a matrix r is the ratio and a matrix t is a matrixobtained from the exposure conditions, and estimates that theeigenvector is the noise components of the images.

(14)

The image processing apparatus according to (13),

wherein the noise components of an image different from an image forwhich the noise components have been calculated are calculated from alinear sum of the estimated noise components.

(15)

The image processing apparatus according to (7),

wherein the estimating unit obtains the noise components for each imageby obtaining g₁, g₂ which satisfy the following formula throughleast-squares estimation,

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack & \; \\{{\begin{bmatrix}t & {- I} \\I & {- r}\end{bmatrix}\begin{bmatrix}g_{1} \\g_{2}\end{bmatrix}} = 0} & \;\end{matrix}$

where r is the ratio, t is the value obtained from the exposureconditions, I is a pixel value of the images and g is the noisecomponents.(16)

An image processing method including:

an estimating step of estimating cyclic noise components included ineach image picked up under different exposure conditions for each image,

wherein the estimating step includes processing of estimating the cyclicnoise components for each image through operation utilizing mutualrelationship between the noise components under the exposure conditions.

(17)

A program causing a computer to execute processing including:

an estimating step of estimating cyclic noise components included ineach image picked up under different exposure conditions for each image,

wherein the estimating step includes processing of estimating the cyclicnoise components for each image through operation utilizing mutualrelationship between the noise components under the exposure conditions.

(18)

Electronic equipment including:

a signal processing unit configured to perform signal processing on apixel signal outputted from an imaging device,

wherein the signal processing unit includes

-   -   an estimating unit configured to estimate cyclic noise        components included in each image picked up under different        exposure conditions for each image, and    -   a correcting unit configured to perform correction to remove        noise from the images using the noise components estimated at        the estimating unit,

wherein the estimating unit estimates the cyclic noise components foreach image through operation utilizing mutual relationship between thenoise components under the exposure conditions.

REFERENCE SIGNS LIST

-   100 imaging apparatus-   101 optical lens-   102 imaging device-   103 image processing unit-   104 signal processing unit-   105 control unit-   311 sensitivity classified interpolating unit-   312 flicker correcting unit-   313 HDR synthesizing unit-   331, 332 extracting unit-   333, 334 interpolation processing unit-   351, 352 exposure correcting unit-   353 blend coefficient calculating unit-   354 blend processing unit-   371 flicker estimating unit-   372 first exposure image flicker correcting unit-   373 second exposure image flicker correcting unit-   401, 402 integrated value calculating unit-   403 dividing unit-   404 estimation operating unit-   431 matrix generating unit-   432 matrix operating unit-   433 flicker component converting unit-   461 Fourier series transforming unit-   462 matrix generating unit-   463 matrix operating unit-   464 Fourier series inverse transforming unit-   465 flicker component converting unit-   466 Fourier series inverse transforming unit-   501, 502 integrated value calculating unit-   503, 504 Fourier series transforming unit-   505 estimation operating unit-   506, 507 Fourier series inverse transforming unit

1. An image processing apparatus comprising: an estimating unitconfigured to estimate cyclic noise components included in each imagepicked up under different exposure conditions for each image, whereinthe estimating unit estimates the cyclic noise components for each imagethrough operation utilizing mutual relationship between the noisecomponents under the exposure conditions.
 2. The image processingapparatus according to claim 1, wherein the cyclic noise is flicker. 3.The image processing apparatus according to claim 1, wherein the mutualrelationship between the noise components is expressed with a shutterfunction of the exposure conditions in frequency space.
 4. The imageprocessing apparatus according to claim 1, wherein the estimating unitestimates the noise components using a value obtained by integrating theimages, multiplying a predetermined window function and performingFourier series expansion.
 5. The image processing apparatus according toclaim 4, wherein the integration is performed in a horizontal directionfor a portion not saturated in any of the images.
 6. The imageprocessing apparatus according to claim 1, wherein the estimating unitobtains the noise components in frequency space by obtaining a matrix Qwhere QF=0 when a fluctuation component of a light source is F andobtaining the fluctuation component F, and wherein the estimating unitestimates the noise components for each image by performing Fourierseries inverse transform on the noise components in the frequency space.7. The image processing apparatus according to claim 1, wherein themutual relationship between the noise components is expressed with aratio obtained by integrating the images and performing division foreach row of the images.
 8. The image processing apparatus according toclaim 7, wherein the integration is performed in a horizontal directionfor a portion not saturated in any of the images.
 9. The imageprocessing apparatus according to claim 7, wherein the estimating unitobtains an eigenvector of an eigenvalue 1 of a matrix RT where R is amatrix obtained by performing Fourier series expansion on the ratio andT is a matrix obtained from the exposure conditions, and sets theeigenvector as a value obtained by performing Fourier series expansionon the noise components of the images.
 10. The image processingapparatus according to claim 9, wherein the noise components of theimages are calculated by performing Fourier series inverse transform onthe eigenvector.
 11. The image processing apparatus according to claim10, wherein a value obtained by performing Fourier series expansion onthe noise components of an image different from an image for which thenoise components have been calculated is calculated by multiplying theeigenvector by a coefficient obtained from the exposure conditions, andwherein the noise components of the images are calculated by performingFourier series inverse transform on the value obtained by performingFourier series expansion.
 12. The image processing apparatus accordingto claim 7, wherein the estimating unit generates a matrix RT in thefollowing formula, $\begin{matrix}{\mspace{79mu} \left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack} \\{{\begin{bmatrix}R_{0} & {\overset{\_}{R}}_{1} & \ldots & {\overset{\_}{R}}_{M} & \; & \; & 0 \\R_{1} & R_{0} & \ddots & \ddots & \ddots & \; & \; \\\vdots & \ddots & R_{0} & {\overset{\_}{R}}_{1} & \ddots & \ddots & \; \\R_{M} & \ddots & \ddots & R_{0} & \ddots & \ddots & {\overset{\_}{R}}_{M} \\\; & \ddots & \ddots & R_{1} & R_{0} & \ddots & \vdots \\\; & \; & \ddots & \ddots & \ddots & R_{0} & {\overset{\_}{R}}_{1} \\0 & \; & \; & R_{M} & \ldots & R_{1} & R_{0}\end{bmatrix}\begin{bmatrix}{\overset{\_}{T}}_{M} & \; & \; & \; & \; & \; & \; \\\; & \ddots & \; & \; & \; & 0 & \; \\\; & \; & {\overset{\_}{T}}_{1} & \; & \; & \; & \; \\\; & \; & \; & T_{0} & \; & \; & \; \\\; & \; & \; & \; & T_{1} & \; & \; \\\; & {0\;} & \; & \; & \; & \ddots & \; \\\; & \; & \; & \; & \; & \; & T_{M}\end{bmatrix}}{\quad{\left\lbrack \begin{matrix}\overset{\_}{G_{1}(M)} \\\vdots \\\overset{\_}{G_{1}(1)} \\{G_{1}(0)} \\{G_{1}(1)} \\\vdots \\{G_{1}(M)}\end{matrix} \right\rbrack = \begin{bmatrix}\overset{\_}{G_{1}(M)} \\\vdots \\\overset{\_}{G_{1}(1)} \\{G_{1}(0)} \\{G_{1}(1)} \\\vdots \\{G_{1}(M)}\end{bmatrix}}}}\end{matrix}$ where R is the ratio, T is the coefficient obtained fromthe exposure conditions and G is the noise components of the images, andobtains the noise components of the images.
 13. The image processingapparatus according to claim 7, wherein the estimating unit obtains aneigenvector of an eigenvalue 1 of a matrix rt where a matrix r is theratio and a matrix t is a matrix obtained from the exposure conditions,and estimates that the eigenvector is the noise components of theimages.
 14. The image processing apparatus according to claim 13,wherein the noise components of an image different from an image forwhich the noise components have been calculated are calculated from alinear sum of the estimated noise components.
 15. The image processingapparatus according to claim 7, wherein the estimating unit obtains thenoise components for each image by obtaining g₁, g₂ which satisfy thefollowing formula through least-squares estimation, $\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack & \; \\{{\begin{bmatrix}t & {- I} \\I & {- r}\end{bmatrix}\begin{bmatrix}g_{1} \\g_{2}\end{bmatrix}} = 0} & \;\end{matrix}$ where r is the ratio, t is the value obtained from theexposure conditions, I is a pixel value of the images and g is the noisecomponents.
 16. An image processing method comprising: an estimatingstep of estimating cyclic noise components included in each image pickedup under different exposure conditions for each image, wherein theestimating step includes processing of estimating the cyclic noisecomponents for each image through operation utilizing mutualrelationship between the noise components under the exposure conditions.17. A program causing a computer to execute processing comprising: anestimating step of estimating cyclic noise components included in eachimage picked up under different exposure conditions for each image,wherein the estimating step includes processing of estimating the cyclicnoise components for each image through operation utilizing mutualrelationship between the noise components under the exposure conditions.18. Electronic equipment comprising: a signal processing unit configuredto perform signal processing on a pixel signal outputted from an imagingdevice, wherein the signal processing unit includes an estimating unitconfigured to estimate cyclic noise components included in each imagepicked up under different exposure conditions for each image, and acorrecting unit configured to perform correction to remove noise fromthe images using the noise components estimated at the estimating unit,wherein the estimating unit estimates the cyclic noise components foreach image through operation utilizing mutual relationship between thenoise components under the exposure conditions.